The mechanism of dynamic force modes has been successfully applied to many atomic force microscopy (AFM) applications, such as tapping mode and phase imaging. The high-order flexural vibration modes are recent advancement of AFM dynamic force modes. AFM optical lever detection sensitivity plays a major role in dynamic force modes because it determines the accuracy in mapping surface morphology, distinguishing various tip-surface interactions, and measuring the strength of the tip-surface interactions. In this work, we have analyzed optimization and calibration of the optical lever detection sensitivity for an AFM cantilever-tip ensemble vibrating in high-order flexural modes and simultaneously experiencing a wide range and variety of tip-sample interactions. It is found that the optimal detection sensitivity depends on the vibration mode, the ratio of the force constant of tip-sample interactions to the cantilever stiffness, as well as the incident laser spot size and its location on the cantilever. It is also found that the optimal detection sensitivity is less dependent on the strength of tip-sample interactions for high-order flexural modes relative to the fundamental mode, i.e., tapping mode. When the force constant of tip-sample interactions significantly exceeds the cantilever stiffness, the optimal detection sensitivity occurs only when the laser spot locates at a certain distance from the cantilever-tip end. Thus, in addition to the “globally optimized detection sensitivity,” the “tip optimized detection sensitivity” is also determined. Finally, we have proposed a calibration method to determine the actual AFM detection sensitivity in high-order flexural vibration modes against the static end-load sensitivity that is obtained traditionally by measuring a force-distance curve on a hard substrate in the contact mode.