Purpose Exploration of the effect of chronic recurrent seizures in focal epilepsy on brain volumes has produced many conflicting reports. To determine differences in brain structure in temporal lobe epilepsy (TLE) and extratemporal epilepsy (using frontal lobe epilepsy (FLE) a surrogate) further, we performed a retrospective analysis of a large cohort of patients with seizure-onset zone proven by intracranial monitoring. Methods A total of 120 TLE patients, 86 FLE patients, and 54 healthy controls were enrolled in this study. An analysis of variance of voxel-based morphometry (VBM) was used to seek morphometric brain differences among TLE patients, FLE patients, and healthy controls. Additionally, a vertex-based surface analysis was utilized to analyze the hippocampus and thalamus. Significant side-specific differences in hippocampal gray matter volume were present between the left TLE (LTLE), right TLE RTLE (RTLE), and control groups ( p<0.05, family-wise error (FWE) corrected). Results Vertex analyses revealed significant volume reduction in inferior parts of the left hippocampus in the LTLE group and lateral parts of the right hippocampus in the RTLE group compared to controls ( p<0.05, FWE corrected). Significant differences were also detected between the LTLE and control group in the bilateral medial and inferior thalamus ( p<0.05, FWE corrected). FLE patients did not exhibit focal atrophy of gray matter across the brain. Conclusion Our results highlight the variation in morphometric lateralized changes in the brain between different epilepsy onset zones, providing critical insight into the natural history of people with drug-resistant focal epilepsies.