Neural systems are necessarily the adaptive products of natural selection, but a neural system, dedicated to any particular function in a complex brain, may be composed of components that covary with functionally unrelated systems, owing to constraints beyond immediate functional requirements. Some studies support a modular or mosaic organization of the brain, whereas others emphasize coordination and covariation. To contrast these views, we have analysed the retina, striate cortex (V1) and extrastriate cortex (V2, V3, MT, etc.) in 30 mammals, examining the area of the neocortex and individual neocortical areas and the relative numbers of rods and cones. Controlling for brain size and species relatedness, the sizes of visual cortical areas (striate, extrastriate) within the brains of nocturnal and diurnal mammals are not statistically different from one another. The relative sizes of all cortical areas, visual, somatosensory and auditory, are best predicted by the total size of the neocortex. In the sensory periphery, the retina is clearly specialized for niche. New data on rod and cone numbers in various New World primates confirm that rod and cone complements of the retina vary substantially between nocturnal and diurnal species. Although peripheral specializations or receptor surfaces may be highly susceptible to niche-specific selection pressures, the areal divisions of the cerebral cortex are considerably more conservative.