The Mindful Attention Awareness Scale (MAAS) is one of the most popular measures of mindfulness, exhibiting promising psychometric properties and theoretically consistent relationships to brain activity, mindfulness-based intervention (MBI) outcomes, and mediation of MBI effects. The present study investigated the response patterns and scale properties in a large sample of undergraduate students (N = 414) using Item Response Theory analyses. The findings suggest that general statements of “automatic inattentiveness” or “automatic pilot” confer greater statistical information about the underlying latent trait. Evidence of limited abilities to report on mindlessness and of response bias to “mindfulness-absent” items suggests challenges to the construct validity of the MAAS. The current findings, along with pre-existing data, suggest that reverse-scoring the scale may be inadequate to represent intentional attention or awareness. Further research is needed to determine which variations, components, and correlates of the numerous operationalizations of mindfulness are theoretically consistent and most salient to positive outcomes, especially in psychopathology.