Purpose To clarify the bereaved family's perceptions about the appropriateness of timing when physicians first referred patients to palliative care units, and to identify the factors contributing to family-perceived late referrals. Subjects and Methods A multicenter questionnaire survey was conducted on 630 bereaved family members of cancer patents who were admitted to palliative care units in Japan. A total of 318 responses were analyzed (effective response rate, 62%). Results Half of the bereaved family members regarded the timing of referrals to palliative care units as late or very late, while less than 5% of families reported early referrals (very late [19%, n = 59], late [30%, n = 96], appropriate [48%, n = 151], early [1.6%, n = 5], and very early [2.2%, n = 7]). Multiple regression analyses revealed that the independent determinants of family-perceived late referrals were: family belief before admission that palliative care shortens the patient's life, insufficient in-advance discussion about preferred end-of-life care between patients/families and physicians, families' insufficient preparation for changes of patient conditions, and hospital admission before referrals. Conclusion In Japan, the timing of referrals to palliative care units was late or very late from the families' perspectives. The independent determinants of family-perceived late referrals were: family misconception about palliative care, inadequate communication with physicians, and families' insufficient preparation for deterioration of patients' conditions. Systematic strategies to overcome these barriers would contribute to providing appropriate palliative care at all stages of cancer.