In this paper, we experimentally study the influence of large-scale Taylor rolls on the small-scale statistics and the flow organization in fully turbulent Taylor–Couette flow for Reynolds numbers up to $Re_{S}=3\times 10^{5}$. The velocity field in the gap confined by coaxial and independently rotating cylinders at a radius ratio of $\unicode[STIX]{x1D702}=0.714$ is measured using planar particle image velocimetry in horizontal planes at different cylinder heights. Flow regions with and without prominent Taylor vortices are compared. We show that the local angular momentum transport (expressed in terms of a Nusselt number) mainly takes place in the regions of the vortex in- and outflow, where the radial and azimuthal velocity components are highly correlated. The efficient momentum transfer is reflected in intermittent bursts, which becomes visible in the exponential tails of the probability density functions of the local Nusselt number. In addition, by calculating azimuthal energy co-spectra, small-scale plumes are revealed to be the underlying structure of these bursts. These flow features are very similar to the one observed in Rayleigh–Bénard convection, which emphasizes the analogies of these systems. By performing a complex proper orthogonal decomposition, we remarkably detect azimuthally travelling waves superimposed on the turbulent Taylor vortices, not only in the classical but also in the ultimate regime. This very large-scale flow pattern, which is most pronounced at the axial location of the vortex centre, is similar to the well-known wavy Taylor vortex flow, which has comparable wave speeds, but much larger azimuthal wavenumbers.