AbstractA diverse set of white matter connections supports seamless transitions between cognitive states. However, it remains unclear how these connections guide the temporal progression of large-scale brain activity patterns in different cognitive states. Here, we analyze the brain’s trajectories across a set of single time point activity patterns from functional magnetic resonance imaging data acquired during the resting state and an n-back working memory task. We find that specific temporal sequences of brain activity are modulated by cognitive load, associated with age, and related to task performance. Using diffusion-weighted imaging acquired from the same subjects, we apply tools from network control theory to show that linear spread of activity along white matter connections constrains the probabilities of these sequences at rest, while stimulus-driven visual inputs explain the sequences observed during the n-back task. Overall, these results elucidate the structural underpinnings of cognitively and developmentally relevant spatiotemporal brain dynamics.