AbstractBecause of the small size of nanomechanical systems, their vibrations become nonlinear already for small amplitudes. Many nontrivial aspects of the vibration dynamics arise from the coexistence of several nonlinearly coupled modes. We show that such coupling can lead to anomalous decay of the modes where they go through nonlinear resonance, so that their amplitude-dependent frequencies become commensurate. We demonstrate the possibility of a strongly nonmonotonic dependence of the decay rate on the amplitude if one of the modes serves as a thermal reservoir for another mode. Where the decay of both modes is slow compared to the rate of resonant energy exchange, the decay is accompanied by amplitude oscillations. Depending on the initial conditions, with increasing time it can display an extremely sharp or a comparatively smooth crossover between different regimes. The results provide insight into recent experimental results by several groups and suggest new ways of characterizing and controlling nanomechanical systems.