The modern olive industry is increasingly interested in olive cultivars that start producing early and remain relatively small, because they are suitable for super high-density orchards. Some cultivars are better suited to this than others but it is not clear why. Understanding the mechanisms that lead to early production and reduced canopy size is therefore important. The object of this study was to investigate whether differences in vigor across olive cultivars are related to earliness and abundance of bearing. We analyzed tree growth and productivity in young coetaneous trees of 12 olive cultivars, grown together in the same orchard. Trunk diameter increased over the observation period, reaching significantly different values across cultivars. Canopy volume also increased, reaching 2-fold differences between the minimum and the maximum values. Cumulative yield increased, reaching up to 3-fold differences. When the cumulative yield at the end of the experiment was plotted against the final trunk diameter, no correlation was found. A significant correlation was found when cumulative yield was plotted against the increment in trunk diameter during the observation period for which yield data were collected. This relationship improved (i.e., R² rose from 0.57 to 0.83) when yield efficiency [i.e., cumulative yield per unit of final trunk cross-sectional area (TCSA) or per unit of canopy volume] was used instead of yield. These results clearly showed that trees that produced proportionally more (i.e., higher yield efficiencies) grew less. We conclude that, in young olive trees, vigor is inversely related to early bearing efficiency, which differs significantly across cultivars. The results support the hypothesis that early and abundant bearing is a major factor in explaining differences in vigor across olive cultivars.