Infants with critical congenital heart defects (CCHDs) are at increased risk for neurodevelopmental delays. The early identification of motor delays is clinically relevant to prevent or reduce long-term consequences. The current study aims to describe the motor-developmental pathways of infants with a CCHD. Motor development was assessed in 215 infants and toddlers using the Dutch version of the Bayley-III. At 3 months (n = 165), 9 months (n = 188), and 18 months (n = 171) the motor composite scores were 97, 98, and 104, respectively. A motor composite score of ≤−2 SD was only seen in 2.4%, 0%, and 2.3%, respectively, with gross motor deficits being observed more often than fine motor deficits (12% vs. 0% at 18 months). Over 90% of infants who scored average at 9 months still did so at 18 months. The majority of infants with below-average gross motor scores (≤−1) at 9 months still had a below-average or delayed motor score (≤−2 SD) at 18 months. Abnormal gross motor scores (≤−2 SD) increased with age. Infants with single-ventricle physiology performed significantly (p ≤ 0.05) worse on both fine and gross motor skills at 9 and 18 months compared to infants with other CCHDs.