We present the first photometric redshift distribution for a large unbiased sample of 870um selected submillimeter galaxies (SMGs) with robust identifications based on observations with the Atacama Large Millimeter Array (ALMA). In our analysis we consider 96 SMGs in the Extended Chandra Deep Field South, 77 of which have 4-19 band, optical-near-infrared, photometry. We model the Spectral Energy Distributions (SEDs) for these 77 SMGs, deriving a median photometric redshift of z=2.3+/-0.1. The remaining 19 SMGs have insufficient optical or near-infrared photometry to derive photometric redshifts, but a stacking analysis of IRAC and Herschel observations confirms they are not spurious. Assuming these sources have an absolute H-band magnitude distribution comparable to that of a complete sample of z~1-2 SMGs, we demonstrate that the undetected SMGs lie at higher redshifts, raising the median redshift for SMGs to z=2.5+/-0.2. More critically we show that the proportion of galaxies undergoing an SMG phase at z>3 is 35+/-5% of the total population. We derive a median stellar mass for SMGs of Mstar=(8+/-1)x10^10Mo, but caution that there are significant systematic uncertainties in our stellar mass estimate, up to x5 for individual sources. We compare our sample of SMGs to a volume-limited, morphologically classified sample of ellipticals in the local Universe. Assuming the star formation activity in SMGs has a timescale of ~100Myr we show that their descendants at z~0 would have a space density and M_H distribution which are in good agreement with those of local ellipticals. In addition the inferred mass-weighted ages of the local ellipticals broadly agree with the look-back times of the SMG events. Taken together, these results are consistent with a simple model that identifies SMGs as events that form most of the stars seen in the majority of luminous elliptical galaxies at the present day. ; Comment: Accepted by ApJ. 45 pages, 16 figures