The rapid modulation of chromatin organization is thought to play a critical role in cognitive processes such as memory consolidation. This is supported in part by the dysregulation of many chromatin remodeling proteins in neurodevelopmental and psychiatric disorders. A key example is ATRX, an X-linked gene commonly mutated in individuals with syndromic and non-syndromic intellectual disability (ID). The consequences of Atrx inactivation on learning and memory have been difficult to evaluate due to the early lethality of hemizygous-null animals. In this study we evaluated the outcome of brain-specific Atrx deletion in heterozygous female mice. The latter exhibit a mosaic pattern of ATRX protein expression in the CNS due to the location of the gene on the X chromosome. While the hemizygous male mice die soon after birth, heterozygous females survive to adulthood. Body growth is stunted in these animals and they have low circulating levels of insulin growth factor 1 (IGF-1). In addition, they are impaired in spatial, contextual fear, and novel object recognition memory. Our findings demonstrate that mosaic loss of ATRX expression in the CNS leads to endocrine defects, decreased body size and has a negative impact on learning and memory.