There has been growing interest in the determinants of the annual timing of biological phenomena, or phenology, in wild populations, but research on vertebrate taxa has primarily focused on the phenology of reproduction. We present here analyses of the phenology of the annual growth of a secondary sexual characteristic, antlers in red deer (Cervus elaphus) males. The long-term individual-based data from a wild population of red deer on the Isle of Rum, Scotland allow us to consider ecological factors influencing variation in the phenology of growth of antlers, and the implications of variation in antler growth phenology with respect to the phenotype of antler grown (antler mass) and annual breeding success. The phenology of antler growth was influenced by local environmental conditions: higher population density delayed both the start date (during spring) and the relative end date (in late summer) of antler growth, and warmer temperatures in the September and April prior to growth advanced start and end dates, respectively. Furthermore, there was variation between individuals in this phenotypic plasticity of start date, although not in that of end date of growth. The phenology of antler growth impacted on the morphology of antlers grown, with individuals who started and ended growth earliest having the heaviest antlers. The timing of antler growth phenology was associated with breeding success in the following mating season, independently of the mass of antlers grown: an earlier start of antler growth was associated with siring a higher number of the calves born the following spring. Our results suggest that the phenology of traits that are not directly correlated with offspring survival may also regularly show correlations with fitness.