AbstractGenetic and hormonal factors have been suggested to influence human sexual orientation. Previous studied proposed brain differences related to sexual orientation and that these follow cross‐sex shifted patterns. However, the neurobiological correlates of sexual orientation and how genetic factors relate to brain structural variation remains largely unexplored. Using the largest neuroimaging‐genetics dataset available on same‐sex sexual behavior (SSB) (n = 18,645), we employed a data‐driven multivariate classification algorithm (PLS) on magnetic resonance imaging data from two imaging modalities to extract brain covariance patterns related to sex. Through analyses of latent variables, we tested for SSB‐related cross‐sex shifts in such patterns. Using genotype data, polygenic scores reflecting the genetic predisposition for SSB were computed and tested for associations with neuroimaging outcomes. Patterns important for classifying between males and females were less pronounced in non‐heterosexuals. Predominantly in non‐heterosexual females, multivariate brain patterns as represented by latent variables were shifted toward the opposite sex. Complementary univariate analyses revealed region specific SSB‐related differences in both males and females. Polygenic scores for SSB were associated with volume of lateral occipital and temporo‐occipital cortices. The present large‐scale study demonstrates multivariate neuroanatomical correlates of SSB, and tentatively suggests that genetic factors related to SSB may contribute to structural variation in certain brain structures. These findings support a neurobiological basis to the differences in human sexuality.