Studying blindness with various onset ages may elucidate the ways that unimodal sensory deprivation at different periods of development shape the human brain. In order to determine the effect of the onset age on brain anatomical networks, we extended a previous study of 17 early blind (EB) subjects with an additional 97 subjects with various onset ages. We constructed binary anatomical networks of these subjects and sighted controls (SC) using diffusion tensor tractography and calculated the topological properties of the network. Based on onset age, the subjects were divided into congenitally blind (CB), EB, adolescent-blind (AB), and late-blind (LB) subgroups. The LB subjects demonstrated a greater connectivity density and a higher global efficiency, similar to the SC. The CB and EB subgroups showed large group differences from the other groups in their topological networks, specifically, a reduced connectivity density and a decreased global efficiency compared with the SC, especially in the frontal and occipital cortices. Additionally, significant correlations were found between age of onset and the topological properties of the anatomical network in the blind. Our results suggest that visual experience during an early period of development is critical for establishing an intact efficient anatomical network in the human brain.