Advances in neuroimaging techniques have recently provided glimpse into the neurobiology of complex traits of human personality. Whereas some intriguing findings have connected aspects of personality to variations in brain morphology, the relations are complex and our current understanding is incomplete. Therefore, we aimed to provide a comprehensive investigation of brain-personality relations using a multimodal neuroimaging approach in a large sample comprising 265 healthy individuals. The NEO Personality Inventory was used to provide measures of core aspects of human personality, and imaging phenotypes included measures of total and regional brain volumes, regional cortical thickness and arealization, and diffusion tensor imaging indices of white matter (WM) microstructure. Neuroticism was the trait most clearly linked to brain structure. Higher neuroticism including facets reflecting anxiety, depression and vulnerability to stress was associated with smaller total brain volume, widespread decrease in WM microstructure, and smaller frontotemporal surface area. Higher scores on extraversion were associated with thinner inferior frontal gyrus, and conscientiousness was negatively associated with arealization of the temporoparietal junction. No reliable associations between brain structure and agreeableness and openness, respectively, were found. The results provide novel evidence of the associations between brain structure and variations in human personality, and corroborate previous findings of a consistent neuroanatomical basis of negative emotionality.