Climatic changes impact fruit tree growth and severely limit their production. Investigating the tree ability to cope with environmental variations is thus necessary to adapt breeding and management strategies in order to ensure sustainable production. In this study, we assessed the genetic parameters and genotype by environment interaction (GxE) during the early tree growth. One hundred and twenty olive seedlings derived from the cross ‘Olivière’ x ‘Arbequina’ were examined across two sites with contrasted environments, accounting for ontogenetic trends over three years. Models including the year of growth, branching order, environment, genotype effects, and their interactions were built with variance function and covariance structure of residuals when necessary. After selection of a model, broad sense heritabilities were estimated. Despite strong environmental effect on most traits, no GxE was found. Moreover, the internal structure of traits co-variation was similar in both sites. Ontogenetic growth variation, related to (i) the overall tree form and (ii) the growth and branching habit at growth unit scale, was not altered by the environment. Finally, a moderate to strong genetic control was identified for traits at the whole tree scale and at internode scale. Among all studied traits, the maximal internode length exhibited the highest heritability (H2 = 0.74). Considering the determinant role of this trait in tree architecture and its stability across environments, this study consolidates its relevance for breeding.