We aimed to evaluate the association between smoking, changes in smoking, and quality of life in a cohort of Spanish university graduates. Smoking habits were self-reported at baseline and four years later. Quality of life was assessed using the Short Form-36 (SF-36) at year 4. Statistical differences in SF-36 scores between groups were determined using ANCOVA with age and sex as covariates. Out of 5,234 eligible participants over 2000–2006, there were 2,639 non-smoker participants, 1,419 ex-smokers, and 1,048 smokers. Within the previous four years, 435 participants became recent quitters and 205 starters. Comparing smoking and health status in year 4, non-smokers showed better scores than the other categories of ever smoking in all dimensions except in the vitality scale value, which was similar in non-smokers and in those smoking less than 15 cigarettes/day. Comparing changes in smoking and health in year 4, continuing smokers had statistically significant worse scores than non-smokers in general health, social functioning, role-emotional and mental health, whereas recent quitters showed statistically significant improvements in role-emotional and mental health over those who had continued smoking or those who became smokers. Our findings support a dose-response relationship between cigarette consumption and a worse quality of life in general and mental health in particular. They also support that changes in smoking have an impact on health.