[abridged] We present observations of the dust and atomic gas phase in seven dwarf irregular galaxies of the M81 group from the SINGS and THINGS surveys. The Spitzer observations provide a first glimpse of the nature of the non-atomic ISM in these metal-poor (Z~0.1 Z_sun), quiescent (SFR~0.001-0.1 M_sun/yr) dwarf galaxies. Dust emission is detected in five out of the seven targets. Most detected dust emission is restricted to HI column densities >1x10^21 cm^-2. Spitzer spectroscopy of two regions in the brightest galaxies (IC 2574 and Holmberg II) show distinctly different spectral shapes. The spectrum of IC 2574 shows aromatic features that are less luminous (relative to the FIR luminosity) compared to an average SINGS spiral galaxy by a factor of \~7 . The aromatic features in Holmberg~II (which has only a slightly lower gas-phase metallicity) are fainter than in IC 2574 by an order of magnitude. This result emphazises that the strength of the aromatic features is not a simple linear function of metallicity. We estimate dust masses of ~10^4-10^6 M_sun for the M81 dwarf galaxies, resulting in an average dust--to--gas ratio (M_dust/M_HI) of ~3x10^-4 (1.5x10^-3 if only the HI that is associated with dust emission is considered); this is an order of magnitude lower than the typical value derived for the SINGS spirals. The dwarf galaxies are underluminous per unit star formation rate at 70um as compared to the more massive galaxies in SINGS by a factor of ~2. However, the average 70um/160um ratio in the sample dwarf galaxies is higher than what is found in the other galaxies of the SINGS sample. This can be explained by a combination of a lower dust content in conjunction with a higher dust temperature in the dwarfs.