We present an experimental and theoretical study of the magnetic field dependence of the mode frequency of thermally excited spin waves in rectangular shaped nanopillars of lateral sizes 60x100, 75x150, and 105x190 nm2, patterned from MgO-based magnetic tunnel junctions. The spin wave frequencies were measured using spectrally resolved electrical noise measurements. In all spectra, several independent quantized spin wave modes have been observed and could be identified as eigenexcitations of the free layer and of the synthetic antiferromagnet of the junction. Using a theoretical approach based on the diagonalization of the dynamical matrix of a system of three coupled, spatially confined magnetic layers, we have modeled the spectra for the smallest pillar and have extracted its material parameters. The magnetization and exchange stiffness constant of the CoFeB free layer are thereby found to be substantially reduced compared to the corresponding thin film values. Moreover, we could infer that the pinning of the magnetization at the lateral boundaries must be weak. Finally, the interlayer dipolar coupling between the free layer and the synthetic antiferromagnet causes mode anticrossings with gap openings up to 2 GHz. At low fields and in the larger pillars, there is clear evidence for strong non-uniformities of the layer magnetizations. In particular, at zero field the lowest mode is not the fundamental mode, but a mode most likely localized near the layer edges. ; Comment: 16 pages, 4 figures, (re)submitted to PRB