We present an analysis of far-infrared (FIR) dust emission from diffuse cirrus clouds. This study is based on serendipitous observations at 160 $mu$m at high-galactic latitude with the Multiband Imaging Photometer onboard the Spitzer Space Telescope by the Spitzer Infrared Nearby Galaxies Survey. These observations are complemented with IRIS data at 100 and 60 $mu$m and constitute one of the most sensitive and unbiased samples of FIR observations at a small scale of diffuse interstellar clouds. Outside regions dominated by the cosmic infrared background fluctuations, we observe a substantial scatter in the 160/100 colors from cirrus emission. We compared the 160/100 color variations to 60/100 colors in the same fields and find a trend of decreasing 60/100 with increasing 160/100. This trend cannot be accounted for by current dust models by changing solely the interstellar radiation field. It requires a significant change of dust properties such as grain size distribution or emissivity or a mixing of clouds in different physical conditions along the line of sight. These variations are important as a potential confusing foreground for extragalactic studies. This work is based on observations made with the Spitzer Space Telescope, which is operated by the Jet Propulsion Laboratory, California Institute of Technology, under a contract with NASA.