The microscopic origin of the magnetically-driven ferroelectricity in collinear AFM-E orthorhombic manganites is explained by means of first-principles Wannier functions. We show that the polarization is mainly determined by the "asymmetric electron hopping" of orbitally-polarized Mn eg states, implicit in the peculiar in-plane zig-zag spin arrangement in the AFM-E configuration. In ortho-HoMnO3, Wannier function centers are largely displaced with respect to corresponding ionic positions, implying that the final polarization is strongly affected by a purely electronic contribution, at variance with standard ferroelectrics where the ionic-displacement is dominant. However, the final value of the polarization is the result of competing effects, as shown by the opposite signs of the contributions to the polarization coming from the Mn eg and t2g states. Furthermore, a systematic analysis of the link between ferroelectricity and the spin, orbital and lattice degrees of freedom in the manganite series has been carried out, in the aim of ascertaining chemical trends as a function of the rare-earth ion. Our results show that the Mn-O-Mn angle is the key quantity in determining the exchange coupling: upon decreasing the Mn-O-Mn angle, the first- (second-) nearest neighbor ferromagnetic (antiferromagnetic) interaction decreases (remains constant), in turn stabilizing either the AFM-A or the AFM-E spin configuration for weakly or strongly distorted manganites, respectively. The Mn eg contribution to the polarization dramatically increases with the Mn-O-Mn angle and decreases with the "long" Mn-O bond length, whereas the Mn t2g contribution decreases with the "short" Mn-O bond-length, partially cancelling the former term. Comment: 11 pages, 10 figures. Submitted for publication