Holly (Ilex L.), from the monogeneric Aquifoliaceae, is a woody dioecious genus cultivated as pharmaceutical and culinary plants, ornamentals, and industrial materials. With distinctive leaf morphology and growth habitats, but uniform reproductive organs (flowers and fruits), the evolutionary relationships of Ilex remain an enigma. To date, few contrast analyses have been conducted on morphology and molecular patterns in Ilex. Here, the different phenotypic traits of four endemic Ilex species (I. latifolia, I. suaveolens, I. viridis, and I. micrococca) on Mount Huangshan, China, were surveyed through an anatomic assay and DNA image cytometry, showing the unspecified link between the examined morphology and the estimated nuclear genome size. Concurrently, the newly-assembled plastid genomes in four Ilex have lengths ranging from 157,601 bp to 157,857 bp, containing a large single-copy (LSC, 87,020–87,255 bp), a small single-copy (SSC, 18,394–18,434 bp), and a pair of inverted repeats (IRs, 26,065–26,102 bp) regions. The plastid genome annotation suggested the presence of numerable protein-encoding genes (89–95), transfer RNA (tRNA) genes (37–40), and ribosomal RNA (rRNA) genes (8). A comprehensive comparison of plastomes within eight Ilex implicated the conserved features in coding regions, but variability in the junctions of IRs/SSC and the divergent hotspot regions potentially used as the DNA marker. The Ilex topology of phylogenies revealed the incongruence with the traditional taxonomy, whereas it informed a strong association between clades and geographic distribution. Our work herein provided novel insight into the variations in the morphology and phylogeography in Aquifoliaceae. These data contribute to the understanding of genetic diversity and conservation in the medicinal Ilex of Mount Huangshan.