We present an analysis of the deepest Herschel images in four major extragalactic fields GOODS-North, GOODS-South, UDS and COSMOS obtained within the GOODS-Herschel and CANDELS-Herschel key programs. The picture provided by 10497 individual far-infrared detections is supplemented by the stacking analysis of a mass-complete sample of 62361 star-forming galaxies from the CANDELS-HST H band-selected catalogs and from two deep ground-based Ks band-selected catalogs in the GOODS-North and the COSMOS-wide fields, in order to obtain one of the most accurate and unbiased understanding to date of the stellar mass growth over the cosmic history. We show, for the first time, that stacking also provides a powerful tool to determine the dispersion of a physical correlation and describe our method called "scatter stacking" that may be easily generalized to other experiments. We demonstrate that galaxies of all masses from z=4 to 0 follow a universal scaling law, the so-called main sequence of star-forming galaxies. We find a universal close-to-linear slope of the logSFR-logM* relation with evidence for a flattening of the main sequence at high masses (log(M*/Msun) > 10.5) that becomes less prominent with increasing redshift and almost vanishes by z~2. This flattening may be due to the parallel stellar growth of quiescent bulges in star-forming galaxies. Within the main sequence, we measure a non varying SFR dispersion of 0.3 dex. The specific SFR (sSFR=SFR/M*) of star-forming galaxies is found to continuously increase from z=0 to 4. Finally we discuss the implications of our findings on the cosmic SFR history and show that more than 2/3 of present-day stars must have formed in a regime dominated by the main sequence mode. As a consequence we conclude that, although omnipresent in the distant Universe, galaxy mergers had little impact in shaping the global star formation history over the last 12.5 Gyr. ; Comment: 30 pages, 26 figures, accepted for publication in A&A