In this paper, we elucidate the optical response resulting from the interplay of charge distribution (ionicity) and Peierls instability (dimerization) in the neutral-ionic ferroelectric phase transition of tetrathiafulvalene-chloranil, a mixed-stack quasi-one-dimensional charge-transfer crystal. We present far-infrared reflectivity measurements down to 5cm-1 as a function of temperature above the phase transition (300-82 K). The coupling between electrons and lattice phonons in the pretransitional regime is analyzed on the basis of phonon eigenvectors obtained through quasiharmonic lattice dynamics, combined with the results of exact numerical diagonalization of the one-dimensional Peierls-Hubbard model. The resulting multiphonon Peierls coupling leads upon approaching the phase transition to a progressive shift of spectral weight and of the coupling strength toward the phonons at lower frequencies, ending in a soft-mode behavior only for the lowest-frequency phonon near the transition temperature. Moreover, in the proximity of the phase transition, the lowest-frequency phonon becomes overdamped due to anharmonicity induced by its coupling to electrons. The implications of these findings for the neutral-ionic transition mechanism are discussed.