We describe the solvent-dependent morphology of a conjugated polymer, poly(3-hexylthiophene) (P3HT) interfaced with [6,6]-phenyl-C61-butyric acid methyl ester (PCBM) and the evolution of the resulting inter-connected fibrillar structure (ICFS) in a diffusive bilayer heterojunction (DBHJ). It is found that the fibrillar structure of the P3HT is preferentially developed in a poor solvent and less soluble than an amorphous structure and the ICFS improves the power conversion efficiency (ηp) significantly. Among four different solvents chlorobenzene, dichlorobenzene, chloroform and p-xylene studied, p-xylene results in ηp ≈ 3% whereas chlorobenzene yields only 0.5%. This indicates clearly that the formation of the ICFS plays a critical role to induce high exciton generation/dissociation to produce spatially uninterrupted pathways for the charge transport, and to enhance ηp of a polymer photovoltaic device in the DBHJ configuration.