The purpose of the study was to assess the association between physical fitness and the lifestyle determinants of elite junior table tennis players. The basic anthropometric characteristics (body height and body weight) were collected of 87 Polish table tennis players (girls, n = 38 and boys, n = 49, at different stages of sport training, targeted and specialized) aged 11–17 years. The level of special fitness tests from the Table Tennis Specific Battery Test were used, assessing reaction speed and displacement speed. All eight International Physical Fitness Test trials were also used to determine the level of general fitness of the participants. Selected questions from the Health Behaviour in School-Aged Children questionnaire were asked to measure factors associated with leisure time. The findings confirm a relationship between sedentary forms of leisure time activity and the training of young players at the targeted stage (Z = −2.93, p = 0.003 school days and Z = −2.12, p = 0.034 days off). Moreover, competitors with longer training experience more often chose active forms of spending free time. Knowledge of the global physical activity undertaken by young athletes during their leisure time provides a better understanding of their individual needs and may help young table tennis players to succeed at a world-class level in the future.