(1) Background: Despite the benefits of physical activity being well documented in university students, some do not follow the international recommendations. This period of life is a vital stage in adhering to healthy habits in adult life. The objective of the study was to analyze university students’ scores of their physical self-concept and its relationship with gender, physical condition, and level of self-perceived competence. (2) Methods: The sample comprised of 1289 Bizkaia University students (42.12% men and 57.87% women), between 18 and 46.5 years old (M = 20.4; SD = 2.2 years). Physical self-concept, physical condition, number of hours per week of physical activity, and perceived fitness level were analyzed. (3) Results: We found significant differences between women and men in their physical self-concept, but it seemed to be mainly an indirect effect through the mediation of hours of exercise and physical condition. (4) Conclusions: To understand the variance of the level of physical self-concept between genders in university students, the effect of certain variables (physical condition, number of hours per week, and perceived fitness level) must be considered, as well as the mediating role of some of these variables.