Cell adhesion and the adhesion of vesicles to the membranes of cells or organelles are pivotal for immune responses, tissue formation, and cell signaling. The adhesion processes depend sensitively on the binding constant of the membrane-anchored receptor and ligand proteins that mediate adhesion, but this constant is difficult to measure in experiments. We have investigated the binding of membrane-anchored receptor and ligand proteins with molecular dynamics simulations. We find that the binding constant of the anchored proteins strongly decreases with the membrane roughness caused by thermally excited membrane shape fluctuations on nanoscales. We present a theory that explains the roughness dependence of the binding constant for the anchored proteins from membrane confinement and that relates this constant to the binding constant of soluble proteins without membrane anchors. Because the binding constant of soluble proteins is readily accessible in experiments, our results provide a useful route to compute the binding constant of membrane-anchored receptor and ligand proteins.