We have run direct N-body simulations to investigate the impact of stellar evolution and dynamics on the structural properties of young massive (3x10^4 solar masses) star clusters (SCs) with different metallicities (Z=1, 0.1, 0.01 solar metallicity). Metallicity drives the mass loss by stellar winds and supernovae (SNe), with SCs losing more mass at high metallicity. We have simulated three sets of initial conditions, with different initial relaxation timescale. We find that the evolution of the half-mass radius of SCs depends on how fast two-body relaxation is with respect to the lifetime of massive stars. If core collapse is slow in comparison with stellar evolution, then mass loss by stellar winds and SNe is the dominant mechanism driving SC evolution, and metal-rich SCs expand more than metal-poor ones. In contrast, if core collapse occurs on a comparable timescale with respect to the lifetime of massive stars, then SC evolution depends on the interplay between mass loss and three-body encounters: dynamical heating by three-body encounters (mass loss by stellar winds and SNe) is the dominant process driving the expansion of the core in metal-poor (metal-rich) SCs. As a consequence, the half-mass radius of metal-poor SCs expands more than that of metal-rich ones. We also find core radius oscillations, which grow in number and amplitude as metallicity decreases. ; Comment: 12 pages, 9 figures, 1 table, MNRAS, accepted