Recently we reported that the topographic organization of visual field eccentricity in human visual cortex extends into high-order, ventral occipitotemporal (VOT) cortex. Within this cortex, regions that respond preferentially to faces and buildings have specific eccentricity biases, suggesting that this category-eccentricity association may reflect differential needs of recognition processes. However, it is still not clear to what extent this center/periphery differentiation within high-order occipitotemporal cortex depends on immediate, moment-to-moment, task demands. Previous attention studies were confined either to exploring the visual field topography (spatial attention) or to object identity (object-based attention). Here, we combined the investigation of these two different attentional mechanisms in the same study. We found that the main source of attentional modulation in occipitotemporal cortex was object-based attention. Shifting attention to different object categories (buildings, faces, and arrows) substantially modulated the object-related activations. The differential activation to each object category in occipitotemporal object areas was maintained, albeit at a reduced level, even when attention was focused on different spatial locations. A slight eccentricity-related attentional differentiation was observed in the more dorsal lateral occipital region, but not in the VOT cortex. These results argue against the possibility that the source of the eccentricity differentiation in VOT cortex is due solely to moment-to-moment shifts in spatial attention mechanism and supports the notion that the eccentricity-biased maps found in this region are due to built-in shape selectivity established over long-term processes.