We investigate the influence of exciton-phonon coupling on the dynamics of a strongly coupled quantum dot-photonic crystal cavity system and explore the effects of this interaction on different schemes for non-classical light generation. By performing time-resolved measurements, we map out the detuning-dependent polariton lifetime and extract the spectrum of the polariton-to-phonon coupling with unprecedented precision. Photon-blockade experiments for different pulse-length and detuning conditions (supported by quantum optical simulations) reveal that achieving high-fidelity photon blockade requires an intricate understanding of the phonons' influence on the system dynamics. Finally, we achieve direct coherent control of the polariton states of a strongly coupled system and demonstrate that their efficient coupling to phonons can be exploited for novel concepts in high-fidelity single photon generation.