Charge transport dynamics in solar cell devices based on as-spun and annealed P3HT:PCBM films are compared using ultrafast time-resolved optical probing of the electric field by means of field-induced second harmonic generation. The results show that charge carriers drift about twice as far during the first 3 ns after photogeneration in a device where the active layer has been thermally annealed. The carrier dynamics were modelled using Monte-Carlo simulations and good agreement between experimental and simulated drift dynamics was obtained using identical model parameters for both cells, but with different average PCBM and polymer domain sizes. The calculations suggest that small domain sizes in as-spun samples limit the carrier separation distance disabling their escape from geminate recombination.