Saffron spice has been used for decades as an ingredient in many dairy products but changes in its coloring properties related to milk characteristics have not been paid appropriate attention. Saffron color was studied in ewes’ milk at different fat levels and saffron concentrations using tristimulus colorimetry. In order to evaluate saffron extraction, different temperatures and extraction times were tested. Color changes were demonstrated to be statistically significant when increasing the fat content in milk, as well as saffron concentration. The higher milk fat content, turned the extracts brighter and yellower, while less red and vivid, opposite to results obtained by increasing saffron concentration. Extraction time was not significant for color extraction. Milk extracts resulted slightly brighter and yellower when increasing temperature, probably due to crocetin esters degradation or isomerization from trans to cis configuration. Temperatures between 37 and 70 °C are recommended to avoid structural changes in milk or saffron. Color changes could be due to interactions mediated by phospholipids between milk fat globules and crocetin esters, as well as minor saffron carotenoids.