Characteristics of the neighborhood built environment influence health and health behavior. Google Street View (GSV) images may facilitate measures of the neighborhood environment that are meaningful, practical, and adaptable to any geographic boundary. We used GSV images and computer vision to characterize neighborhood environments (green streets, visible utility wires, and dilapidated buildings) and examined cross-sectional associations with chronic health outcomes among patients from the University of California, San Francisco Health system with outpatient visits from 2015 to 2017. Logistic regression models were adjusted for patient age, sex, marital status, race/ethnicity, insurance status, English as preferred language, assignment of a primary care provider, and neighborhood socioeconomic status of the census tract in which the patient resided. Among 214,163 patients residing in California, those living in communities in the highest tertile of green streets had 16–29% lower prevalence of coronary artery disease, hypertension, and diabetes compared to those living in communities in the lowest tertile. Conversely, a higher presence of visible utility wires overhead was associated with 10–26% more coronary artery disease and hypertension, and a higher presence of dilapidated buildings was associated with 12–20% greater prevalence of coronary artery disease, hypertension, and diabetes. GSV images and computer vision models can be used to understand contextual factors influencing patient health outcomes and inform structural and place-based interventions to promote population health.