We propose a new method of pushing $Herschel$ to its faintest detection limits using universal trends in the redshift evolution of the far infrared over 24$μ$m colours in the well-sampled GOODS-North field. An extension to other fields with less multi-wavelength information is presented. This method is applied here to raise the contribution of individually detected $Herschel$ sources to the cosmic infrared background (CIRB) by a factor 5 close to its peak at 250$μ$m and more than 3 in the 350$μ$m and 500$μ$m bands. We produce realistic mock $Herschel$ images of the deep PACS and SPIRE images of the GOODS-North field from the GOODS-$Herschel$ Key Program and use them to quantify the confusion noise at the position of individual sources, i.e., estimate a "local confusion noise". Two methods are used to identify sources with reliable photometric accuracy extracted using 24$μ$m prior positions. The clean index (CI), previously defined but validated here with simulations, which measures the presence of bright 24$μ$m neighbours and the photometric accuracy index (PAI) directly extracted from the mock $Herschel$ images. After correction for completeness, thanks to our mock $Herschel$ images, individually detected sources make up as much as 54% and 60% of the CIRB in the PACS bands down to 1.1 mJy at 100$μ$m and 2.2 mJy at 160$μ$m and 55, 33, and 13% of the CIRB in the SPIRE bands down to 2.5, 5, and 9 mJy at 250$μ$m, 350$μ$m, and 500$μ$m, respectively. The latter depths improve the detection limits of $Herschel$ by factors of 5 at 250$μ$m, and 3 at 350$μ$m and 500$μ$m as compared to the standard confusion limit. Interestingly, the dominant contributors to the CIRB in all $Herschel$ bands appear to be distant siblings of the Milky Way ($z$$∼$0.96 for $λ$$