Unavoidable variations in size and position of the building blocks of photonic crystals cause light scattering and extinction of coherent beams. We present a model for both two- and three-dimensional photonic crystals that relates the extinction length to the magnitude of the variations. The predicted lengths agree well with our experiments on high-quality opals and inverse opals, and with literature data analyzed by us. As a result, control over photons is limited to distances up to 50 lattice parameters (~15 um) in state-of-the-art structures, thereby impeding applications that require large photonic crystals, such as proposed optical integrated circuits. Conversely, scattering in photonic crystals may lead to different physics such as Anderson localization and nonclassical diffusion.