Purpose This prospective longitudinal study evaluated the effect of androgen deprivation therapy (ADT) on objective and self-reported physical function. Patients and Methods Men with nonmetastatic prostate cancer (PC) starting continuous ADT were enrolled in this matched cohort study. Physical function was assessed with the 6-minute walk test (6MWT), grip strength, and the timed-up-and-go (TUG) test, representing endurance and upper and lower extremity strength, respectively. Quality of life (QOL) was measured with the Medical Outcomes Study Short-Form 36 (SF-36) questionnaire. Subjects were assessed at baseline, 3, 6, and 12 months. Two control groups (PC without ADT; no PC), matched on age, education, and baseline function were enrolled. Mixed effects regression models were fitted, adjusting for baseline covariates. Results We enrolled 87 patients on ADT, 86 PC controls, and 86 healthy controls; groups were similar in age (mean, 69.1 years; range, 50 to 87) and physical function. The 6MWT distance remained stable in the ADT group (P = .96) but improved in both control groups (P < .05). Grip strength declined in the ADT group (P = .04), remained stable in PC controls (P = .31), and improved in healthy controls (P = .008). TUG scores remained stable over time and across groups (P > .10). The SF-36 physical function summary score declined in the ADT group (P < .001), but increased in both control groups (P < .001). Negative effects on outcomes were observed within 3 months of starting ADT and were generally independent of age. Conclusion Endurance, upper extremity strength, and physical components of QOL are affected within 3 months of starting ADT. Up-front exercise interventions to counteract these losses are warranted.