Background: Living at moderate altitude may be associated with health benefits, including reduced mortality from male colorectal and female breast cancer. We aimed to determine altitude-dependent incidence and mortality rates of those cancers and put them in the context of altitude-associated lifestyle differences. Methods: Incidence cases and deaths of male colorectal cancer (n = 17,712 and 7462) and female breast cancer (n = 33,803 and 9147) from altitude categories between 250 to about 2000 m were extracted from official Austrian registries across 10 years (2008–2017). Altitude-associated differences in health determinants were derived from the Austrian Health Interview Survey (2014). Results: The age-standardized incidence and mortality rates of male colorectal cancer decreased by 24.0% and 44.2%, and that of female breast cancer by 6.5% and 26.2%, respectively, from the lowest to the highest altitude level. Higher physical activity levels and lower body mass index for both sexes living at higher altitudes were found. Conclusions: Living at a moderate altitude was associated with a reduced incidence and (more pronounced) mortality from colorectal and breast cancer. Our results suggest a complex interaction between specific climate conditions and lifestyle behaviours. These observations may, in certain cases, support decision making when changing residence.