The supercontinuum generated with a linearly polarized near-IR (775 nm) pump in rotated calcium fluoride is shown to have intrinsic intensity and polarization modulations. To mask the rotation of the crystal plate, we circularly polarize the pump and find greatly improved output parameters for the generated white light: intensity fluctuations of 0.5% limited only by pump laser stability, and a circular polarization state-matching that of the pump-over the entire visible spectrum. This polarization conservation allows the return of the supercontinuum to a linear polarization state or to a pair of linearly polarized beams with correlated intensity fluctuations. We were also able to extend the supercontinuum source deep into the ultraviolet with a frequency doubled (387 nm) pump, to serve as an new source to probe the region where most molecular photochemistry occurs.