Background The ratings of physician-patient communication are an important indicator of the quality of health care delivery and provide guidance for many important decisions in the health care setting and in health research. But there is no gold standard to assess physician-patient communication. Thus, depending on the specific measurement condition, multiple sources of variance may contribute to the total score variance of ratings of physician-patient communication. This may systematically impair the validity of conclusions drawn from rating data. Objective To examine the extent to which different measurement conditions and rater perspectives, respectively contribute to the variance of physician-patient communication ratings. Methods Variance components of ratings of physician-patient communication gained from 32 general practitioners and 252 patients from 25 family practices in Germany were analyzed using generalizability theory. The communication dimensions “shared decision making”, “effective and open communication” and “satisfaction” were considered. Results Physician-patient communication ratings most substantially reflect unique rater-perspective and communication dimension combinations (32.7% interaction effect). The ratings also represented unique physician and rater-perspective combinations (16.3% interaction effect). However, physicians’ communication behavior and the observed communication dimensions revealed only a low extent of score variance (1% physician effect; 3.7% communication dimension effect). Approximately half of the variance remained unexplained (46.2% three-way interaction, confounded with error). Conclusion The ratings of physician-patient communication minimally reflect physician communication skills in general. Instead, these ratings exhibit primarily differences among physicians and patients in their tendency to perceive shared decision making and effective and open communication and to be satisfied with communication, regardless of the communication behavior of physicians. Rater training and assessing low inferential ratings of physician-patient communication dimensions should be considered when subjective aspects of rater perspectives are not of interest.