This cross-sectional study with 825 students from fifth to ninth grade ( M_age = 13.01, SD = 1.30) investigated how self-concept domains (physical fitness and verbal) are associated with general self-esteem. Extending the internal/external frame of reference model, school grades in physical education and German (as a subject) were considered as correlates of the verbal and physical fitness self-concept domains. In addition, considering the well-established empirical evidence on gender differences in the different subdomains, measurement and structural invariance were tested across female and male students. Grades, self-concept, and self-esteem were assessed via questionnaires. Structural equation modeling revealed positive relations between grades and the corresponding self-concept subdomains. Physical self-concept had a stronger association with general self-esteem compared with verbal self-concept. Gender stereotypical differences were found only in the mean values of self-concept subdomains and general self-esteem. The results indicate that school grades and the corresponding self-concepts are strongly associated and that physical self-concept might be a salient and meaningful source of general self-esteem in adolescents.