Linear birefringence and dichroism of magnetite ferrofluids are studied simultaneously using spectroscopic ellipsometry in transmission mode. It is shown that this versatile technique enables highly accurate characterisation of magneto-optical phenomena. Magnetic field-dependent linear birefringence and dichroism as well as the spectral dependence are shown to be in line with previous results. Despite the qualitative agreement with established models for magneto-optical phenomena, these fail to provide an accurate, quantitative description of our experimental results using the bulk dielectric function of magnetite. We discuss the results in relation to these models, and indicate how the modified dielectric function of the magnetite nanoparticles can be obtained.