Context. How the quiescent galaxies evolve with redshift and the factors that impact their evolution are still debated. It is still unclear what the dominant mechanisms of passive galaxy growth are and what role is played by the environment in shaping their evolutionary paths over cosmic time. Aims. The population of quiescent galaxies is altered over time by several processes that can affect their mean properties. Our aim is to study the mass–size relation (MSR) of the quiescent population and to understand how the environment shapes the MSR at intermediate redshift. Methods. We used the VIMOS Public Extragalactic Redshift Survey (VIPERS), a large spectroscopic survey of ∼90 000 galaxies in the redshift range 0.5 ≤ z ≤ 1.2. We selected a mass-complete sample of 4786 passive galaxies based on the NUVrK diagram and refined it using the D_n4000 spectral index to study the MSR of the passive population over 0.5 ≤ z ≤ 0.9. The impact of the environment on the MSR and on the growth of the quiescent population is studied through the density contrast. Results. The slope and the intercept of the MSR, α = 0.62 ± 0.04 and log(A) = 0.52 ± 0.01, agree well with values from the literature at the same redshift. The intercept decreases with redshift, R_e(z) = 8.20 × (1 + z)^−1.70, while the slope remains roughly constant, and the same trend is observed in the low-density (LD) and high-density (HD) environments. Thanks to the largest spectroscopic sample at 0.5 ≤ z ≤ 0.9, these results are not prone to redshift uncertainties from photometric measurements. We find that the average size of the quiescent population in the LD and HD environments are identical within 3σ and this result is robust against a change in the definition of the LD and HD environments or a change in the selection of quiescent galaxies. In the LD and HD environments, ∼30 and ∼40% of the population have experienced a minor merger process between 0.5 ≤ z ≤ 0.9. However, minor mergers account only for 30–40% of the size evolution in this redshift range, the remaining evolution likely being due to the progenitor bias.