We present a novel system to calibrate millimeter-wave polarimeters for cosmic microwave background (CMB) polarization measurements. This technique is an extension of the conventional metal mirror rotation approach, however, it employs cryogenically-cooled blackbody absorbers. The primary advantage of this system is that it can generate a slightly polarized signal (∼100 mK) in the laboratory; this is at a similar level to that measured by ground-based CMB polarization experiments observing a ∼10 K sky. It is important to reproduce the observing condition in the laboratory for reliable characterization of polarimeters before deployment. In this paper, we present the design and principle of the system and demonstrate its use with a coherent-type polarimeter used for an actual CMB polarization experiment. This technique can also be applied to incoherent-type polarimeters and it is very promising for the next-generation CMB polarization experiments.