Previous research suggests that within Mandarin-speaking congenital amusics, only a subgroup has behavioral lexical tone perception impairments (tone agnosia), whereas the rest of amusics do not. The purpose of the current study was to investigate the categorical nature of lexical tone perception in Mandarin-speaking amusics with and without behavioral lexical tone deficits. Three groups of listeners (controls, pure amusics, and amusics with tone agnosia) participated in tone identification and discrimination tasks. Indexes of the categorical perception (CP) of a physical continuum of fundamental frequencies ranging from a rising to level tone were measured. Specifically, the stimulus durations were manipulated at 100 and 200 ms. For both stimulus durations, all groups exhibited similar categorical boundaries. The pure amusics showed sharp identification slopes and significantly peaked discrimination functions similar to those of normal controls. However, such essential characteristics for the CP of lexical tones were not observed in amusics with tone agnosia. An enlarged step-size from 20 to 35 Hz was not able to produce any discrimination peaks in tone agnosics either. The current study revealed that only amusics with tone agnosia showed a lack of categorical tone perception, while the pure amusics demonstrated typical CP of lexical tones, indicating that the deficit of pitch processing in music does not necessarily result in the deficit in the CP of lexical tones. The different performance between congenital amusics with and without tone agnosia provides a new perspective on the proposition of the relationship between music and speech perception.