In eukaryotes, the genetic material is stored in the nucleus, which is enclosed in a double lipid bilayer, the nuclear envelope (NE). It protects the genome from physical stress and separates it from the rest of the cell. On top of this physical function, growing evidence shows that the nuclear periphery contributes to the 3D organization of the genome. In turn, tridimensional organization of chromatin in the nuclear space influences genome expression. Here we review recent findings on the function of this physical barrier in gene repression and latest models on how silent subnuclear compartments at the NE are built in yeast as well as in the nematode C. elegans and mammalian cells; trying to draw parallels between the three systems.