Possible violations of Lorentz invariance (LIV) can produce vacuum birefringence, which results in a frequency-dependent rotation of the polarization plane of linearly polarized light from distant sources. In this paper, we try to search for a frequency-dependent change of the linear polarization angle arising from vacuum birefringence in the spectropolarimetric data of astrophysical sources. We collect five blazars with multiwavelength polarization measurements in different optical bands (UBVRI). Taking into account the observed polarization angle contributions from both the intrinsic polarization angle and the rotation angle induced by LIV, and assuming that the intrinsic polarization angle is an unknown constant, we obtain new constraints on LIV by directly fitting the multiwavelength polarimetric data of the five blazars. Here, we show that the birefringence parameter η quantifying the broken degree of Lorentz invariance is limited to be in the range of −9.63×10−8<η<6.55×10−6 at the 2σ confidence level, which is as good as or represents one order of magnitude improvement over the results previously obtained from ultraviolet/optical polarization observations. Much stronger limits can be obtained by future multiwavelength observations in the gamma-ray energy band.