Few studies have explored the inter-relationships of sources of social support and caregiving self-efficacy with caregiver burden and patient’s quality of life among patients with palliative care needs and their caregivers. This study tested the associations of two sources of social support (family and friends) and the mediating role of caregiving self-efficacy on caregiver burden and patient’s quality of life. A convenience sample of 225 patient–caregiver dyads recruited between September 2016 and May 2017 from three hospitals in Hong Kong was included in the current analysis. Results showed that the final model provided a satisfactory fit (SRMR = 0.070, R-RMSEA = 0.055 and R-CFI = 0.926) with the data, as good as the hypothesized model did (p = 0.326). Significant associations were detected. Family support had a significant negative indirect effect on caregiver burden and a significant positive indirect effect on patient’s quality of life through caregiving self-efficacy, whereas friend support had a significant positive direct effect on caregiver burden but a minimal effect, if any, on patient’s quality of life. These findings emphasized (1) the importance of caregiving self-efficacy in improving caregiver burden and patient’s quality of life and that (2) sources of social support may be an important dimension moderating the associations of caregiving self-efficacy with caregiver burden and patient’s quality of life.