Four different kinds of photovoltaic polymers were synthesized by controlling the intrachain noncovalent coulomb interactions through the incorporation of alkoxy-or alkylthio-substituted phenylene, 4,7-di(furan-2-yl)benzothiadiazole, and 4,7-di(thiophen-2-yl)benzothiadiazole as a building block. Fine modulation of the interplay of dipole−dipole, H-bond, and chalcogen−chalcogen interactions (O···S, O···H, S···S, S···F, etc.) along the polymeric backbone influenced the chain planarity, interchain organization, film morphology, and electrical and photovoltaic properties significantly. By replacing the alkoxy substituents with alkylthio groups, the torsional angle increased (136−168°) due to the absence of an O···S attractive coulomb interaction (and/or increased S···S steric hindrance), enhancing the amorphous nature with hindered interchain packing. The alkoxy-substituted polymers exhibited nanofibrillar structures, showing strong interlamellar scattering peaks up to (300) with tight face-on π−π stacking in grazing incidence X-ray scattering. The measured carrier mobility of the alkoxy-containing polymers was 1−2 orders of magnitude higher than that of the alkylthio-containing polymers. The incident-light-intensity-dependent photovoltaic characteristics clearly suggested efficient charge generation/extraction with less charge recombination for the alkoxy-containing semicrystalline polymers. The resulting photovoltaic energy conversion efficiency of the PPDT2FBT, PPDF2FBT, PPsDF2FBT, and PPsDT2FBT blended devices with PC 70 BM was measured to be 8.28%, 5.63%, 5.12%, and 0.55%, respectively. This study suggests an important molecular design guideline for the further optimization of photovoltaic polymers and devices by finely controlling the interplay of the weak noncovalent coulomb interactions.