Little is known about the impact of congenital Zika virus (ZIKV) exposure on growth in the first years of life. In this prospective cohort study,201 ZIKV antenatally-exposed children were followed at a tertiary referral center in Rio de Janeiro, Brazil. Eighty-seven were classified as congenital Zika syndrome (CZS) patients and 114 as not congenital Zika syndrome (NCZS); growth parameters were described and compared between groups and with WHO standard growth curves. Thirty-four (39%) newborns with CZS and seven (6%) NCZS were small for gestational age (p < 0.001). NCZS mean weight measures ranged from −0.45 ± 0.1 to 0.27 ± 0.2 standard deviations (SD) from the WHO growth curve median during follow-up, versus −1.84 ± 0.2 to −2.15 ± 0.2 SD for the CZS group (p < 0.001). Length mean z-scores varied from −0.3 ± 0.1 at 1 month to 0.17 ± 0.2 SD between 31 and 36 months in the NCZS group, versus −2.3 ± 0.3 to −2.0 ± 0.17 SD in the CZS group (p < 0.001). Weight/height (W/H) and BMI z-scores reached -1.45 ± 0.2 SD in CZS patients between 31 and 36 months, versus 0.23 ± 0.2 SD in the NCZS group (p < 0.01). Between 25 and 36 months of age, more than 50% of the 70 evaluated CZS children were below weight and height limits; 36 (37.1%) were below the W/H cut-off. Gastrostomy was performed in 23 (26%) children with CZS. During the first three years of life, CZS patients had severe and early growth deficits, while growth of NCZS children was normal by WHO standards.