AbstractAccurate estimation of gross primary productivity (GPP) is essential for understanding the terrestrial carbon budget. Current large‐scale GPP estimates are often obtained at coarse resolutions without considering the subpixel heterogeneity, leading to scaling errors in results. Here, to further characterize (a) the critical sub‐upscaling process causing the largest error and (b) the contributions of various heterogeneity factors in causing the scaling errors, a hydrology‐vegetation model was used to estimate GPP at the 30 m resolution (assumed as reality), and other coarser resolutions (60, 120, 240, 480, and 960 m, assumed as approximations) for 16 mountainous watersheds. Then, GPP scaling errors in the upscaling process of surface heterogeneity were investigated by the root mean squared error between the reality and approximations. Results showed that any surface heterogeneity aggregation from fine to coarse resolutions (e.g., 30–960 m) could cause GPP scaling errors (133 ± 40 gCm⁻²yr⁻¹), and the aggregation from medium to coarse resolutions (e.g., 240–960 m) may be the largest source. More specifically, GPP scaling errors caused by the vegetation heterogeneity aggregation from fine to medium resolutions were relatively small, and the GPP errors caused by the surface topography aggregation from fine to coarse resolutions were all non‐negligible. Elevation aggregation caused larger GPP scaling error than the aggregations of land cover, leaf area index, slope, and aspect. This work highlights the need to consider surface heterogeneity (especially the elevation information) when modeling mountain vegetation GPP at coarse resolutions.