The association between visual impairment and higher mortality remains unclear. In addition, evidence is lacking on the interaction between visual function and physical activity on mortality. We used data of individuals with no disability or with visual impairment among those who participated in the National Health Screening Program in Korea in 2009 or 2010. We constructed Cox proportional hazard models adjusted for potential confounders to evaluate the independent association between visual impairment and mortality. More severe visual impairment was associated with higher all-cause mortality (p-value for trend = 0.03) and mortality due to cardiovascular diseases (p-value for trend = 0.02) and that due to other diseases (p-value for trend = 0.01). We found an interaction on an additive scale between visual impairment and no physical activity on all-cause mortality (relative excess risk due to interaction = 1.34, 95% confidence interval: 0.37, 2.30, p-value = 0.01). When we stratified the study population by physical activity, the association between visual impairment and mortality was only found among individuals who did not engage in regular physical activity (p-value for trend = 0.01). We found an independent association between visual impairment and mortality and modification of this association by physical activity.