Using ultrafast polarization-controlled transient absorption (TA) measurements, dynamics of the initial exciton states were investigated on the time scale of tens of femtoseconds to about 80 picoseconds in two different types of conjugated polymers extensively used in active layers of organic photovoltaic devices. These polymers are poly(3-fluorothienothiophenebenzodithiophene) (PTB7) and poly-3-hexylthiophene (P3HT), which are charge-transfer polymers and homopolymers, respectively. In PTB7, the initial excitons with excess vibrational energy display two observable ultrafast time constants, corresponding to coherent exciton diffusion before the vibrational relaxation, and followed by incoherent exciton diffusion processes to a neighboring local state after the vibrational relaxation. In contrast, P3HT shows only one exciton diffusion or conformational motion time constant of 34 ps, even though its exciton decay kinetics are multi-exponential. Based on the experimental results, an exciton dynamics mechanism is conceived taking into account the excitation energy and structural dependence in coherent and incoherent exciton diffusion processes, as well as other possible deactivation processes including the formation of the pseudo-charge-transfer and charge separate states in polymer chains.