Recent observational results obtained with SCUBA, COBE and ISO have greatly improved our knowledge of the infrared and sub-mm background radiation. These limits become constraining given the realization that most AGNs are heavily obscured and must reradiate strongly in the IR/sub-mm. Here we predict the contribution of AGNs to the IR/sub-mm background, starting from measurements of the hard X-ray background. We show that an application of what we know of AGN Spectral Energy Distributions (SEDs) and the IR background requires that a significant fraction of the 10-150 micron background comes from AGNs. This conclusion can only be avoided if obscured AGNs are intrinsically brighter in the X-rays (with respect to the optical-UV) than unobscured AGNs, contrary to ``unified schemes'' for AGNs, or have a dust to gas ratio much lower (< 0.1) than Galactic. We show that these results are rather robust and not strongly dependent on the details of the modeling. Comment: 13 pages, 1 figure, Astrophysical Journal, in press