The morphology of the active layer plays a crucial role in determining device performance and stability for organic solar cells. All-polymer solar cells (All-PSCs), showing robust and stable morphologies, have been proven to give better thermal stability than their fullerene counterparts. However, outstanding thermal stability is not always the case for polymer blends, and the limiting factors responsible for the poor thermal stability in some All-PSCs, and how to obtain higher efficiency without losing stability, still remain unclear. By studying the morphology of poly [2,3-bis (3-octyloxyphenyl) quinoxaline-5,8-diyl-alt-thiophene-2,5-diyl](TQ1)/poly[4,8-bis[5-(2-ethylhexyl)-2-thienyl]benzo[1,2-b:4,5-b′]dithiophene-alt-(4-(2-ethylhexyl)-3-fluorothieno[3,4-b]thiophene-)-2-carboxylate-2-6-diyl]] (PCE10)/PNDI-T10 blend systems, we found that the rearranged molecular packing structure and phase separation were mainly responsible for the poor thermal stability in devices containing PCE10. The TQ1/PNDI-T10 devices exhibited an improved PCE with a decreased π–π stacking distance after thermal annealing; PCE10/PNDI-T10 devices showed a better pristine PCE, however, thermal annealing induced the increased π–π stacking distance and thus inferior hole conductivity, leading to a decreased PCE. Thus, a maximum PCE could be achieved in a TQ1/PCE10/PNDI-T10 (1/1/1) ternary system after thermal annealing resulting from their favorable molecular interaction and the trade-off of molecular packing structure variations between TQ1 and PCE10. This indicates that a route to efficient and thermal stable All-PSCs can be achieved in a ternary blend by using material with excellent pristine efficiency, combined with another material showing improved efficiency under thermal annealing.