We present an analysis of the Yangian symmetries of various bosonic sectors of the dilatation operator of $\cal N$$=4$ SYM. The analysis is presented from the point of view of Hamiltonian matrix models. In the various SU(n) sectors, we give a modified presentation of the Yangian generators, which are conserved on states of any size. A careful analysis of the Yangian invariance of the full SO(6) sector of the scalars is also presented in this paper. We also study the Yangian invariance of the dilatation operator beyond first order perturbation theory in the SU(2) sector. Following this, we derive the continuum limits of the various matrix models and reproduce the sigma model actions for fast moving strings reported in the recent literature. We motivate the constructions of continuum sigma models (corresponding to both the SU(n) and SO(n) sectors) as variational approximations to the matrix model Hamiltonians. These sigma models retain the semi-classical counterparts of the original Yangian symmetries of the dilatation operator. The semi-classical Yangian symmetries of the sigma models are worked out in detail. The zero curvature representation of the equations of motion and the construction of the transfer matrix for the SO(n) sigma model obtained as the continuum limit of the one loop bosonic dilatation operator is carried out, and the similar constructions for the SU(n) case are also discussed. ; Comment: 40 pages, 5 figures. Typos corrected and references added in revised version