This study deals with how pain characteristics in conjunction with other factors affect quality of life (QoL) in a vulnerable primary care population. We recruited vulnerable older people (75+, n = 825) living in south-eastern Sweden. A postal questionnaire included pain aspects, QoL (EQ-5D-3L, RAND-36 physical functioning, attitudes toward own aging, and life satisfaction), functional status, social networks, and basic demographic information. Pain extent and localization was obtained by digitalization of pain drawings reported on standard body charts. Most respondents were experiencing pain longer than 3 months (88.8%). Pain frequency varied mostly between occasionally (33.8%) and every day (34.8%). A minority reported high pain intensity (13.6%). The lower back and lower legs were the most frequently reported pain locations (>25%). Multiple linear regression model revealed three characteristics of pain (intensity, frequency, and extent) remained inversely associated with the EQ-5D-3L index score (R2 = 0.57). Individually, each of these pain characteristics showed a negative impact on the other three dimensions of QoL (R2 = 0.23–0.59). Different features of pain had impact on different dimensions of QoL in this aging population. A global pain assessment is useful to facilitate individual treatment and rehabilitation strategies in primary care.