Research in both animals and humans shows that some nutrients are important in pregnancy and during the first years of life to support brain and cognitive development. Our aim was to evaluate the role of selenium (Se) in supporting brain and behavioral plasticity and maturation. Pregnant and lactating female rats and their offspring up to postnatal day 40 were fed isocaloric diets differing in Se content—i.e., optimal, sub-optimal, and deficient—and neurodevelopmental, neuroinflammatory, and anti-oxidant markers were analyzed. We observed early adverse behavioral changes in juvenile rats only in sub-optimal offspring. In addition, sub-optimal, more than deficient supply, reduced basal glial reactivity in sex dimorphic and brain-area specific fashion. In female offspring, deficient and sub-optimal diets reduced the antioxidant Glutathione peroxidase (GPx) activity in the cortex and in the liver, the latter being the key organ regulating Se metabolism and homeostasis. The finding that the Se sub-optimal was more detrimental than Se deficient diet may suggest that maternal Se deficient diet, leading to a lower Se supply at earlier stages of fetal development, stimulated homeostatic mechanisms in the offspring that were not initiated by sub-optimal Se. Our observations demonstrate that even moderate Se deficiency during early life negatively may affect, in a sex-specific manner, optimal brain development.