A central finding of functional MRI studies is the highly selective response of distinct brain areas in the occipital temporal cortex to faces and places. However, little is known about the association of white matter fibers with the processing of these object categories. In the current study we used DTI-based tractography to reconstruct two main fibers that connect the occipital lobe with the anterior temporal lobe (inferior longitudinal fasciculus-ILF) and with the frontal lobe (inferior fronto-occipital fasciculus-IFOF) in normal individuals. In addition to MRI scans subjects performed face, scene and body recognition tasks outside the scanner. Results show that recognition of faces and scenes were selectively associated with separate parts of the ILF. In particular, face recognition was highly associated with the fractional anisotropy (FA) of the anterior part of the ILF in the right hemisphere. In contrast, scene recognition was strongly correlated with the FA of the posterior and middle but not the anterior part of the ILF bilaterally. Our findings provide the first demonstration that faces and places are not only associated with distinct brain areas but also with separate parts of white matter fibers.