ImportancePhysicians and medical students who desire to build families face significant barriers due to the structure and culture of medicine.ObjectiveTo understand the barriers and facilitators to family building for all people in medicine—not only individuals who can become pregnant—through an open-ended, qualitative analysis of survey responses.Design, Setting, and ParticipantsThis qualitative study used a survey conducted in April and May 2021 with a broad sample of physicians and medical students. Participants were recruited through social media, targeting physician and medical student communities. Physicians (residents, fellows, and physicians in independent practice) and medical students of all gender identities and sexual orientations were included. Informed by a postpositivist approach, coding reliability thematic analysis was performed on 3 open-ended survey questions on family-building experiences (what they would do differently, what advice they have for others, and anything else they wished to share).Main Outcomes and MeasuresIdentified themes were mapped to the social-ecological model, a model used in public health to examine how a spectrum of factors is associated with health outcomes.ResultsA total of 2025 people (1860 [92%] women; 299 [15%] Asian, 151 [8%] Black, and 1303 [64%] White; 1730 [85%] heterosexual; and 1200 [59%] physicians who had completed training) responded to at least 1 of 3 open-ended questions. Themes mapped to social-ecological model levels included: (1) cultural, eg, medical training being at odds with family building; (2) organizational, eg, lack of institutional support for the range of family-building routes; (3) interpersonal, eg, impact of social support on family building; and (4) individual, eg, socioeconomic status and other individual factors that facilitate or inhibit family building. Recommendations to improve family-building experiences include implementing family-building curricula at medical schools, providing adequate parental leave for all physicians and medical students who become parents, and providing insurance coverage for all family-building routes.Conclusions and RelevanceIn this qualitative study of physicians and medical students, self-reported barriers to family building were identified at each level of the social-ecological model. Addressing these barriers is critical to creating a more equitable family-building environment for physicians and medical students.