Using a phenomenological approach, we self-consistently model the redshift evolution of the ultraviolet (UV) and infrared (IR) luminosity functions across cosmic time, as well as a range of observed IR properties of UV-selected galaxy population. This model is an extension of the 2SFM (2 star-formation modes) formalism, which is based on the observed "main-sequence" of star-forming galaxies, i.e. a strong correlation between their stellar mass and their star formation rate (SFR), and a secondary population of starbursts with an excess of star formation. The balance between the UV light from young, massive stars and the dust-reprocessed IR emission is modeled following the empirical relation between the attenuation (IRX for IR excess hereafter) and the stellar mass, assuming a scatter of 0.4\,dex around this relation. We obtain a good overall agreement with the measurements of the IR luminosity function up to z~3 and the UV luminosity functions up to z~6, and show that a scatter on the IRX-M relation is mandatory to reproduce these observables. We also naturally reproduce the observed, flat relation between the mean IRX and the UV luminosity at L$_{\rm UV}>$10$^{9.5}$ L$_⊙$. Finally, we perform predictions of the UV properties and detectability of IR-selected samples and the vice versa, and discuss the results in the context of the UV-rest-frame and sub-millimeter surveys of the next decade. ; Comment: Accepted for publication in MNRAS; 14 pages; 7 figures