This work focuses on detecting and localizing anomalous events in videos of crowded scenes, i.e. divergences from a dominant pattern. Both motion and appearance information are considered, so as to robustly distinguish different kinds of anomalies, for a wide range of scenarios. A newly introduced concept based on swarm theory, Histograms of Oriented Swarms (HOS), is applied to capture the dynamics of crowded environments. HOS, together with the well known Histograms of Oriented Gradients (HOG), are combined to build a descriptor that effectively characterizes each scene. These appearance and motion features are only extracted within spatiotemporal volumes of moving pixels to ensure robustness to local noise, increase accuracy in the detection of local, nondominant anomalies, and achieve a lower computational cost. Experiments on benchmark datasets containing various situations with human crowds, as well as on traffic data, led to results that surpassed the current state of the art, confirming the method's efficacy and generality. Finally, the experiments show that our approach achieves significantly higher accuracy, especially for pixel-level event detection compared to State of the Art (SoA) methods, at a low computational cost.