We study massive galaxies at z ~ 1-3.5 using HST optical imaging, ground-based near-IR imaging, and Spitzer observations at 3-24 μm. From Ks-selected galaxies in the 130 arcmin2 GOODS-S field, we identify 153 distant red galaxies (DRGs) with (J - Ks)Vega ≥ 2.3. This sample is approximately complete in stellar mass for passively evolving galaxies above 1011 M☉ and z ≤ 3. Roughly half of the DRGs are objects whose optical and near-IR rest-frame light is dominated by evolved stars combined with ongoing star formation (at zmed ~ 2.5), and the others are galaxies whose light is dominated by heavily reddened (A1600 4-6 mag) starbursts (at zmed ~ 1.7). Very few DRGs (10%) have no indication of current star formation. DRGs at z ~ 1.5-3 with stellar masses ≥1011 M☉ have specific star formation rates (SFRs per unit mass) including the reradiated far-IR emission that range from 0.2 to 10 Gyr-1. Based on the X-ray luminosities and rest-frame near-IR colors, roughly one-quarter of the DRGs contain AGNs, implying that the growth of supermassive black holes coincides with the formation of massive galaxies. At 1.5 ≤ z ≤ 3, the DRGs with M ≥ 1011 M☉ have an integrated specific SFR comparable to the global value of all galaxies. In contrast, galaxies at z ~ 0.3-0.75 with M ≥ 1011 M☉ have an integrated specific SFR less than the global value and more than an order of magnitude lower than that for massive DRGs. At z 1, lower mass galaxies dominate the overall cosmic mass assembly. This suggests that the bulk of star formation in massive galaxies occurs at early cosmic epochs and is largely complete by z ~ 1.5. Further mass assembly in these galaxies takes place with low specific SFRs.