Brillouin light scattering shows that shape distortions in Ni80Fe20 nanomagnets can have a dramatic effect on the measured collective linewidth of certain spin-wave modes. The intentional introduction of quantifiable asymmetric egglike shape distortion to an ideal elliptical structure lifts the degeneracy of end modes with concentrated amplitude at the nanomagnet edges. In contrast, modes with concentrated amplitude at the interior are significantly less affected by the distortion. The splitting of end modes by asymmetric distortions explains the large inhomogeneous linewidth broadening in end modes found in large ensembles of nanomagnets that contain a relatively small statistical variation in the degree of distortion.