The primary tumor location (PTL) is associated with the phenotype, metastatic sites, mutations, and outcomes of metastatic colorectal cancer (mCRC) patients, but this has mostly been studied according to sidedness (right vs. left sided). We studied right colon vs. left colon vs. rectal PTL in a real-life study population (n = 1080). Health-related quality of life (HRQoL) was assessed multi-cross-sectionally with QLQ-C30, QLQ-CR29, EQ-5D, and 15D. A chi-square, Kaplan–Meier, and Cox regression were used to compare the groups. The PTL was in the right colon in 310 patients (29%), the left colon in 396 patients (37%), and the rectum in 375 patients (35%). The PTL was associated with distinct differences in metastatic sites during the disease trajectory. The resectability, conversion, and resection rates were lowest in the right colon, followed by the rectum, and were highest in the left colon. Overall survival was shortest for right colon compared with left colon or rectal PTL (median 21 vs. 35 vs. 36 months), with the same trends after metastasectomy or systemic therapy only. PTL also remained statistically significant in a multivariable model. The distribution of symptoms varied according to PTL, especially between the right colon (with general symptoms of metastases) and rectal PTL (with sexual- and bowel-related symptoms). mCRC, according to PTL, behaves differently regarding metastatic sites, resectability of the metastases, outcomes of treatment, and HRQoL.