We use a sample of 37 of the densest clusters and protoclusters across $1.3 \le z \le 3.2$ from the Clusters Around Radio-Loud AGN (CARLA) survey to study the formation of massive cluster galaxies. We use optical $i'$-band and infrared 3.6$μ$m and 4.5$μ$m images to statistically select sources within these protoclusters and measure their median observed colours; $〈 i'-[3.6] 〉$. We find the abundance of massive galaxies within the protoclusters increases with decreasing redshift, suggesting these objects may form an evolutionary sequence, with the lower redshift clusters in the sample having similar properties to the descendants of the high redshift protoclusters. We find that the protocluster galaxies have an approximately unevolving observed-frame $i'-[3.6]$ colour across the examined redshift range. We compare the evolution of the $〈 i'-[3.6] 〉$ colour of massive cluster galaxies with simplistic galaxy formation models. Taking the full cluster population into account, we show that the formation of stars within the majority of massive cluster galaxies occurs over at least 2Gyr, and peaks at $z ∼ 2$-3. From the median $i'-[3.6]$ colours we cannot determine the star formation histories of individual galaxies, but their star formation must have been rapidly terminated to produce the observed red colours. Finally, we show that massive galaxies at $z>2$ must have assembled within 0.5Gyr of them forming a significant fraction of their stars. This means that few massive galaxies in $z>2$ protoclusters could have formed via dry mergers. ; Comment: Accepted for publication in MNRAS. 20 pages including 13 figures & 2 appendices