Apart from the general finding of larger global brain volumes in men, neuroimaging studies that compared brain structure between men and women have yielded some inconsistencies with regard to regional differences. One confound when comparing men and women may be differences in their genetic and or family background. A design that addresses such confounds compares brain structures between brothers and sisters, who share their genetic and family background. In the present study, we aimed to contribute to the existing literature on structural brain sex differences by comparing regional gray and white matter volume, using voxel based morphometry (VBM); and white matter microstructure, using tract-based spatial statistics (TBSS), between 40 unrelated males and females, and contrasting the results with those obtained in a group of 47 opposite-sex siblings, including 42 dizygotic opposite-sex (DOS) twin pairs." Our results showed that men had larger global brain volumes as well as higher mean fractional anisotropy across the brain and showed regionally enlarged gray matter volume and higher fractional anisotropy in, or surrounding, subcortical structures (hypothalamus, thalamus, putamen and globus pallidus and rostral midbrain). Increased gray matter volume in women was restricted to areas of the cortex, including inferior temporal, insular, cingulate, precentral and frontal/prefrontal regions. These sex differences were generally consistent between the unrelated male-female pairs and the opposite-sex sibling pairs. Therefore, we conclude that these sex differences are not the result of confounding differences in genetic or family background and that the etiology of these sex differences merits further investigation.