The way a horse activates (effort phase-EP) and recovers (recovery phase-RP) during a sport event can affect its sport performance. The aim of this manuscript was to test horses’ adaptation to sport performance and its genetic basis, using eye temperature assessed with infrared thermography. EP and RP were measured in 495 Spanish Sport Horses, during a performance test, considering sex (2) and genetic lines (5) as fixed effects. The ranking position obtained on an official sport competition was also collected. Differences in variables due to genetic line and sex effects were found, showing that, regardless of the genetic line, stallions tended to recover better than mares after the sport test developed. High positive intra-class correlations (p < 0.001) were found between EP and RP for both fixed effects, so that the higher the EP, the higher the RP. However, for the ranking position, a low negative correlation (p < 0.01) was found, so that the higher the eye temperature increase, the better the position. Heritabilities showed medium–high values with a medium positive genetic correlation between them. Thus, breed origins and sex influence horses’ effort and recovery during sport performance, showing a genetic basis adequate for selection.