This study examined cerebral asymmetry, especially in the hierarchical visual system, when reading Chinese characters. Twelve right-handed Chinese children (mean age = 11.6 years) were scanned while performing semantic and phonological tasks. Strong leftward asymmetry was found in the left inferior frontal cortex (BA44/45/47), the parietal lobule (BA40), and the cingulate cortex (BA24/32). In the visual system, we found significant left-hemispheric dominance in the fusiform cortex (BA19/37), but no asymmetry was found in the primary visual cortex (BA17/18). The differential results for the primary visual cortex versus high-order visual cortex (i.e., the fusiform cortex) are discussed in terms of the contribution of the logographic nature of Chinese characters to the asymmetry pattern in the hierarchical visual system.