ABSTRACT We derived the dust-to-H i ratio of the intermediate-velocity clouds (IVCs), the high-velocity clouds (HVCs), and the local H i gas, by carrying out a multiple-regression analysis of the 21 cm H i emission combined with the sub-mm dust optical depth. The method covers over 80 per cent of the sky contiguously at a resolution of 47 arcmin and is distinguished from the absorption-line measurements toward bright galaxies and stars covering a tiny fraction of the sky. Major results include that the ratio of the IVCs is in a range of 0.1–1.5 with a mode at 0.6 (relative to the solar-neighbourhood value, likewise below) and that a significant fraction, ∼20 per cent, of the IVCs include dust-poor gas with a ratio of <0.5. It is confirmed that 50 per cent of the HVC Complex C has a ratio of <0.3, and that the Magellanic Stream has the lowest ratio with a mode at ∼0.1. The results prove that some IVCs have low metallicity gas, contrary to the previous absorption-line measurements. Considering that the recent works show that the IVCs are interacting and exchanging momentum with the high-metallicity Galactic halo gas, we argue that the high-metallicity gas contaminates a significant fraction of the IVCs. Accordingly, we argue that the IVCs include a significant fraction of the low-metallicity gas supplied from outside the Galaxy as an alternative to the Galactic-fountain model.