Heating and cooling, as the basic requirements of human life, account for more than 28% of global energy consumption. To minimize carbon footprint and save energy, various strategies based on passive heating and cooling have attracted much attention. Typically, as the sun and outer space are the natural thermodynamic resources for renewable energy harvesting, passive solar heating and radiative cooling have been widely explored through a precise spectral design to effectively exploit solar energy and thermal radiation. In this Perspective, based on the previous works and principal development of passive solar heating and radiative cooling, the corresponding ideal spectral design under different temperature conditions is proposed and highlighted. As demonstrations, we present several examples of different optical spectral designs for realizing passive solar heating and radiative cooling to fit various scenarios. Finally, possible solutions to the current problems and the directions for future development are discussed.