We report on ground and excited state charge transfer in charge-transfer complexes in films formed between a semiconducting polymer, MEH-PPV (poly[2-methoxy-5-(2′-ethylhexyloxy)-1,4-phenylenevinylene]), and a series of fluorene electron acceptors. The latter were designed to vary systematically the electron affinity (EA) over ∼1.5 eV by attachment of various electron withdrawing groups to the fluorene core. The EAs of the acceptors are determined by cyclic voltammetry and compared with those from density functional theory calculations. The charge-transfer dynamics are studied using an ultrafast visible-pump–IR-probe photoinduced absorption technique. We demonstrate that the acceptor EA is the key—but not the only—parameter that governs charge recombination rates that scale exponentially with the acceptor EA. From the time-resolved data we deduced a model that describes charge dynamics for acceptors with low and high EAs. The two opposite trends—higher acceptor EA increases the driving force for charge separation but also inevitably increases the rate of undesirable charge recombination—should be carefully counterbalanced in designing novel polymer–fullerene bulk heterojunctions.