InP-based quantum dots (QDs) have Stokes shifts and photoluminescence (PL) line widths that are larger than in II–VI semiconductor QDs with comparable exciton energies. The mechanisms responsible for these spectral characteristics are investigated in this paper. Upon comparing different semiconductors, we find the Stokes shift decreases in the following order: InP > CdTe > CdSe. We also find that the Stokes shift decreases with core size and decreases upon deposition of a ZnSe shell. We suggest that the Stokes shift is largely due to different absorption and luminescent states in the angular momentum fine structure. The energy difference between the fine structure levels, and hence the Stokes shifts, are controlled by the electron–hole exchange interaction. Luminescence polarization results are reported and are consistent with this assignment. Spectral widths are controlled by the extent of homogeneous and inhomogeneous broadening. We report PL and PL excitation (PLE) spectra that facilitate assessing the roles of homogeneous and different inhomogeneous broadening mechanisms in the spectra of zinc-treated InP and InP/ZnSe/ZnS particles. There are two distinct types of inhomogeneous broadening: size inhomogeneity and core–shell interface inhomogeneity. The latter results in a distribution of core–shell band offsets and is caused by interfacial dipoles associated with In–Se or P–Zn bonding. Quantitative modeling of the spectra shows that the offset inhomogeneity is comparable to but somewhat smaller than the size inhomogeneity. The combination of these two types of inhomogeneity also explains several aspects of reversible hole trapping dynamics involving localized In3+/VZn2− impurity states in the ZnSe shells.