Imaging traits provide a powerful and biologically relevant substrate to examine the influence of genetics on the brain. Interest in genome-wide, brain-wide search for influential genetic variants is growing, but has mainly focused on univariate, SNP-based association tests. Moving to gene-based multivariate statistics, we can test the combined effect of multiple genetic variants in a single test statistic. Multivariate models can reduce the number of statistical tests in gene-wide or genome-wide scans and may discover gene effects undetectable with SNP-based methods. Here we present a gene-based method for associating the joint effect of single nucleotide polymorphisms (SNPs) in 18,044 genes across 31,662 voxels of the whole brain in 731 elderly subjects (mean age: 75.56 ± 6.82SD years; 430 males) from the Alzheimer’s Disease Neuroimaging Initiative (ADNI). Structural MRI scans were analyzed using tensor-based morphometry (TBM) to compute 3D maps of regional brain volume differences compared to an average template image based on healthy elderly subjects. Using the voxel-level volume difference values as the phenotype, we selected the most significantly associated gene (out of 18,044) at each voxel across the brain. No genes identified were significant after correction for multiple comparisons, but several known candidates were re-identified, as were other genes highly relevant to brain function. GAB2, which has been previously associated with late-onset AD, was identified as the top gene in this study, suggesting the validity of the approach. This multivariate, gene-based voxelwise association study offers a novel framework to detect genetic influences on the brain.