In the present study, we aimed to identify the impact of chosen anthropometric measurements on the special physical fitness of elite junior table tennis players at different stages of sport training. A total of 87 table tennis players aged 13.4 ± 1.74 years (43.7% girls and 56.3% boys) from two Polish teams were analyzed. The anthropometry measurements included height, sitting height, body weight, arm span, humerus and femur breadths, five skinfold thicknesses, and five girths were assessed. Participants’ somatotypes were also calculated using the Heath–Carter method as well as body mass index (BMI), which was constructed using the lambda, mu, sigma (LMS) method. Body composition via a bioelectric impedance analysis was also analyzed. The level of special fitness of athletes was determined using tests from the Table Tennis Specific Battery Test, assessing reaction and displacement speeds. Mesomorphic (4.1) and ectomorphic (3.8) profiles were registered for boys and girls, respectively. Boys achieved higher scores than girls for almost all variables, with the exception of ectomorphic somatotype (p = 0.274), skinfold triceps (p = 0.444), and calf skinfold medial (p = 0.609). The relationship between the body height, thickness of the skinfolds of the triceps and suprailiac, biceps, and waist circumference and arm span in all three motor tests was observed, simultaneously significantly higher results were obtained by competitors at the specialist stage of training. Knowledge of the somatic and motor characteristics of young athletes can help coaches in creating a specific training program for improved health and performance, taking into consideration the athletes’ biological development, potential, and pre-disposition.