Background: A multi-disciplinary approach has led to an improvement in prognosis of childhood cancers. However, in parallel with the increase in survival rate, there is a greater occurrence of long-term toxicity related to antineoplastic treatment. Hypogonadism and infertility are among the most frequent endocrinological sequelae in young adult childhood cancer survivors. The aim of this study was to identify which category of patients, grouped according to diagnosis, therapy, and age at treatment, shows the worst reproductive function in adulthood. Methods: We evaluated morpho-volumetric development of the testis, endocrine function of the hypothalamic–pituitary–gonadal axis, and sperm parameters in 102 young adult childhood cancer survivors. Results: Overall, about one-third of patients showed low total testicular volume, total testosterone (TT) <3.5 ng/mL, and altered sperm count. Hodgkin’s disease, hematopoietic stem cell transplantation, and non-cranial irradiation associated to chemotherapy were risk factors for poor gonadal function. Patients treated in pubertal age showed lower total testicular volume; however, the difference was due to more gonadotoxic treatment performed in older age. Testicular volume was more predictive of spermatogenesis than follicle-stimulating hormone (FSH), while anti-Müllerian hormone (AMH) was not useful in the evaluation of testicular function of male childhood cancer survivors. Conclusions: Pre-pubertal subjects at high risk of future infertility should be candidates for testicular tissue cryopreservation.