The urge to move to music is universal among humans. Unlike visual art, which is manifest across space, music is manifest across time. When listeners get carried away by the music, either through movement (such as dancing) or through reverie (such as trance), it is usually the temporal qualities of the music—its pulse, tempo, and rhythmic patterns—that put them in this state. In this article, we review studies addressing rhythm, meter, movement, synchronization, entrainment, the perception of groove, and other temporal factors that constitute a first step to understanding how and why music literally moves us. The experiments we review span a range of methodological techniques, including neuroimaging, psychophysics, and traditional behavioral experiments, and we also summarize the current studies of animal synchronization, engaging an evolutionary perspective on human rhythmic perception and cognition.