We show that impurities and vacancies affect the thermal conductivity much more strongly than what is predicted by widely accepted models. When local distortions around point defects are strong, standard perturbative approaches fail, and phonon scattering can only be accounted for by an exact Green's function calculation. We apply the theory to the study, from first-principles, of nitrogen and vacancy defects in diamond. The thermal conductivity is computed by solving the linearized Boltzmann transport equation. The Born approximation underestimates the phonon scattering cross sections of nitrogen and vacancies by factors of 3 and 10, respectively. Thermal conductivity calculations are in good agreement with experiment.