Electron–photon temporal correlations in electron energy loss spectroscopy (EELS) and cathodoluminescence (CL) spectroscopies have recently been used to measure the relative quantum efficiency of materials. This combined spectroscopy, named cathodoluminescence excitation (CLE) spectroscopy, allows for the identification of excitation and decay channels, which are hidden in average measurements. Here, we demonstrate that CLE can also be used to measure excitations' decay time. In addition, the decay time as a function of the excitation energy is measured, as the energy for each electron–photon pair is probed. We used two well-known insulating materials to characterize this technique, nanodiamonds with NV0 defects and hexagonal boron nitride (h-BN) with 4.1 eV defects. Both also exhibit marked transition radiations, whose extremely short decay times can be used to characterize the instrumental response function. It is found to be typically 2 ns, in agreement with the expected limit of the EELS detector temporal resolution. The measured lifetimes of NV0 centers in diamond nanoparticles (20–40 ns) and 4.1 eV defect in h-BN flakes (<2 ns) match those reported previously.