Abstract Background Brassica oleracea includes several morphologically diverse, economically important vegetable crops, such as the cauliflower and cabbage. However, genetic variants, especially large structural variants (SVs), that underlie the extreme morphological diversity of B. oleracea remain largely unexplored. Results Here we present high-quality chromosome-scale genome assemblies for two B. oleracea morphotypes, cauliflower and cabbage. Direct comparison of these two assemblies identifies ~ 120 K high-confidence SVs. Population analysis of 271 B. oleracea accessions using these SVs clearly separates different morphotypes, suggesting the association of SVs with B. oleracea intraspecific divergence. Genes affected by SVs selected between cauliflower and cabbage are enriched with functions related to response to stress and stimulus and meristem and flower development. Furthermore, genes affected by selected SVs and involved in the switch from vegetative to generative growth that defines curd initiation, inflorescence meristem proliferation for curd formation, maintenance and enlargement, are identified, providing insights into the regulatory network of curd development. Conclusions This study reveals the important roles of SVs in diversification of different morphotypes of B. oleracea, and the newly assembled genomes and the SVs provide rich resources for future research and breeding.