Developing materials with large optical nonlinearity as well as ultrafast optical response is crucial for high-speed integrated photonic devices. Besides large optical nonlinearity at telecommunication wavelengths, some degenerated semiconductors as transparent conductive oxides are found to have subpicosecond optical responses, yet the theoretical elucidation of such unexpected fast temporal dynamics is still lacking. In this study, after resonant intraband excitation, the recovery of transient nonlinear response was revealed to be 20 times faster in indium tin oxide than in gold with ultrafast time-resolved transmission spectroscopy. By simulating the optical response processes using the two-temperature model, we found that the electron–phonon scattering rates of indium tin oxide and aluminum doped zinc oxide are about 2 orders of magnitude as large as that of gold, which is suggested to be the main origin of the diverse optical response speed. This study quantitatively attributes the measured transient optical response to ultrafast quasi-particle interactions and gives new insights into the theoretical description of the ultrafast dynamics in both metals and degenerated semiconductors.