In order to explore the reason why the single-layered cuprates, La$_{2-x}$(Sr/Ba)$_x$CuO$_4$ ($T_c≃$ 40K) and HgBa$_2$CuO$_{4+δ}$ ($T_c≃$ 90K), have such a significant difference in $T_c$, we study a two-orbital model that incorporates the $d_{z^2}$ orbital on top of the $d_{x^2-y^2}$ orbital. It is found, with the fluctuation exchange approximation, that the $d_{z^2}$ orbital contribution to the Fermi surface, which is stronger in the La system, works against d-wave superconductivity, thereby dominating over the effect of the Fermi surface shape. The result resolves the long-standing contradiction between the theoretical results on Hubbard-type models and the experimental material dependence of $T_c$ in the cuprates. ; Comment: 4 pages, 4 figures. Final version published in Phys. Rev. Lett