We investigate the single-mode phonon transmission in symmetry-broken carbon nanotubes with the Stone-Wales defect using a mode-matching method. To gain an insight into the features of the transmission, the symmetry properties of phonons are analyzed within the line group symmetry theory for single-walled carbon nanotubes. Our results show that the transmission for acoustic phonons with low group velocity is heavily suppressed, and the long-wave optical phonons near the Γ point are strongly scattered by defects. We clarify that this reduction in transmission for phonons with small group velocity is a universal behavior resulting from the broken translational invariance. Furthermore, fine structures of dips in the transmission are observed for some phonon modes, depending on the unmatched symmetry operations between the specific phonon mode and the defect. Our findings reveal that heat generation in experiments for the defective nanotubes may be contributed largely from the phonons with low group velocity because of their low transmissions. We propose the mode-dependent transmission for the single-mode phonon be utilized to identify the different phonon modes through the features for the fine structures of the transmission in experiments.