It has been suggested that physical education (PE) teachers hold strong anti-fat attitudes and that these can have an impact on the health of their students. In this study, we compare the attitudes and stereotypes towards obesity of PE teachers with those of their colleagues who teach mathematics. In addition, we evaluated the association between the teachers’ anti-fat biases and the level of physical activity (PA) of their students. The sample consisted of 81 PE teachers and 75 mathematics teachers. The adolescent sample consisted of 1792 secondary school students. The assessment of attitudes and stereotypes was conducted with the Implicit Association Test and the Anti-Fat Attitudes questionnaire. The PA level of the adolescents was determined by a self-administered instrument. PE and mathematics teachers showed similar biases, except for the automatic obesity–laziness association, which was stronger for PE teachers than for mathematics teachers. These distinctive implicit stereotypes of PE teachers were also significant predictors of the lack of PA in adolescents. We recommend the use of interventions aimed at reducing non-traditional forms of prejudice in teachers.