Abstract At present, China’s power system is developing in the direction of AC-DC parallel connection. AC-DC hybrid transmission has become the inevitable pattern of China’s future power grid. In the operation of AC/DC hybrid system, the optimization of multiple indexes is often considered. One of the economic problems of system operation is how to minimize the operation cost of the system. This requires the optimal solution of a system of nonlinear equations with constraints. Among the existing algorithms, the interior point method has excellent convergence performance and has been widely used in power system. In this paper, the basic principle of interior point method is introduced, and the interior point method is used to optimize the AC and DC power flow obtained. With the network operation cost as the objective optimization index. The Matlab software is used for programming and simulation calculation. On the basis of verifying the effectiveness of the algorithm, the algorithm is transplanted into the large network for calculation and analysis, and the adaptability of the test algorithm.