Autism spectrum disorders (ASDs) are a heterogeneous group of neurodevelopmental disorders of genetic and environmental etiologies. Some ASD cases are syndromic: associated with clinically defined patterns of somatic abnormalities and a neurobehavioral phenotype (e.g., Fragile X syndrome). Many cases, however, are idiopathic or non-syndromic. Such disorders present themselves during the early postnatal period when language, speech, and personality start to develop. ASDs manifest by deficits in social communication and interaction, restricted and repetitive patterns of behavior across multiple contexts, sensory abnormalities across multiple modalities and comorbidities, such as epilepsy among many others. ASDs are disorders of connectivity, as synaptic dysfunction is common to both syndromic and idiopathic forms. While multiple theories have been proposed, particularly in idiopathic ASDs, none address why certain brain areas (e.g., frontotemporal) appear more vulnerable than others or identify factors that may affect phenotypic specificity. In this hypothesis article, we identify possible routes leading to, and the consequences of, altered connectivity and review the evidence of central and peripheral synaptic dysfunction in ASDs. We postulate that phenotypic specificity could arise from aberrant experience-dependent plasticity mechanisms in frontal brain areas and peripheral sensory networks and propose why the vulnerability of these areas could be part of a model to unify preexisting pathophysiological theories.