ABSTRACT Models for cosmic ray (CR) dynamics fundamentally depend on the rate of CR scattering from magnetic fluctuations. In the ISM, for CRs with energies ∼MeV-TeV, these fluctuations are usually attributed either to ‘extrinsic turbulence’ (ET) – a cascade from larger scales – or ‘self-confinement’ (SC) – self-generated fluctuations from CR streaming. Using simple analytic arguments and detailed ‘live’ numerical CR transport calculations in galaxy simulations, we show that both of these, in standard form, cannot explain even basic qualitative features of observed CR spectra. For ET, any spectrum that obeys critical balance or features realistic anisotropy, or any spectrum that accounts for finite damping below the dissipation scale, predicts qualitatively incorrect spectral shapes and scalings of B/C and other species. Even if somehow one ignored both anisotropy and damping, observationally required scattering rates disagree with ET predictions by orders of magnitude. For SC, the dependence of driving on CR energy density means that it is nearly impossible to recover observed CR spectral shapes and scalings, and again there is an orders-of-magnitude normalization problem. But more severely, SC solutions with super-Alfvénic streaming are unstable. In live simulations, they revert to either arbitrarily rapid CR escape with zero secondary production, or to bottleneck solutions with far-too-strong CR confinement and secondary production. Resolving these fundamental issues without discarding basic plasma processes requires invoking different drivers for scattering fluctuations. These must act on a broad range of scales with a power spectrum obeying several specific (but plausible) constraints.