Coolia Meunier is an important component of benthic dinoflagellate assemblages in tropical and subtropical seas. In this study, detailed morphological observation of Coolia species from Malaysian waters was carried out using light and electron microscopy in parallel with molecular characterization of nuclear-encoded partial LSU rDNA, and internal transcribed spacer (ITS) regions. Live specimens were collected from seaweed samples and established into clonal cultures. There are significant morphological variations between the Malaysian isolates in comparison to the type species, C. monotis Meunier. The feature that differentiates the new species is the third postcingular plate (3′′′), which is the largest hypothecal plate in the Malaysian isolates, whereas in C. monotis, the 3′′′ and 4′′′ plates are almost equal in size. Detailed observations of the thecal pores also revealed the presence of fine perforations within the pores of the Malaysian isolates, but these perforations are absent in C. monotis. Comparisons between Malaysian isolates and C. monotis nucleotide sequence of the ITS region showed high genetic divergence at 28%, in contrast to the 0.3%–3% divergence observed among populations of the same species. Structural comparison of the second internal transcribed spacer (ITS2) rRNA transcript between the two species showed compensatory base changes (CBCs) in the three helices of ITS2 rRNA. Based on morphological and molecular data, the Malaysian isolates are considered to represent a new species, for which the name Coolia malayensis is proposed.