Calcium aluminate cement (CAC) is less commonly used as ordinary Portland cements (OPCs) for structural concrete, is relatively expensive, but may have certain advantages when used for solidification of wastes; it introduces rapid strength gain and has a higher resistance to chemical attack than OPC. However, the most widely identified degradation process suffered by CAC is the so-called conversion of hexagonal calcium aluminate hydrate to a cubic form. Mixes of CAC with silica fume (SF) or fly ash (FA) represent an interesting alternative for the stabilization of CAC hydrates, which might be attributed to a microstructure based mainly on aluminosilicates. This paper deals with the microstructure of cement pastes fabricated with binders of CAC–SF and CAC–FA, and their evolution over time. Mid infrared and near infrared spectroscopy have been used to assess the microstructure of these formulations. Microstructural characterization was completed by backscattering electron microscopy observation and microanalysis.