Background: Prenatal exposure to cannabis may influence childhood cognition and behavior, but the epidemiologic evidence is mixed. Even less is known about the potential impact of secondhand exposure to cannabis during early childhood. Objective: This study sought to assess whether prenatal and/or postnatal exposure to cannabis was associated with childhood cognition and behavior. Study design: This sub-study included a convenience sample of 81 mother–child pairs from a Colorado-based cohort. Seven common cannabinoids (including delta 9-tetrahydrocannabinol (Δ9-THC) and cannabidiol (CBD)) and their metabolites were measured in maternal urine collected mid-gestation and child urine collected at age 5 years. Prenatal and postnatal exposure to cannabis was dichotomized as exposed (detection of any cannabinoid) and not exposed. Generalized linear models examined the associations between prenatal or postnatal exposure to cannabis with the NIH Toolbox and Child Behavior Checklist T-scores at age 5 years. Results: In this study, 7% (n = 6) of the children had prenatal exposure to cannabis and 12% (n = 10) had postnatal exposure to cannabis, with two children experiencing this exposure at both time points. The most common cannabinoid detected in pregnancy was Δ9-THC, whereas the most common cannabinoid detected in childhood was CBD. Postnatal exposure to cannabis was associated with more aggressive behavior (β: 3.2; 95% CI: 0.5, 5.9), attention deficit/hyperactivity problems (β: 8.0; 95% CI: 2.2, 13.7), and oppositional/defiant behaviors (β: 3.2; 95% CI: 0.2, 6.3), as well as less cognitive flexibility (β: −15.6; 95% CI: −30.0, −1.2) and weaker receptive language (β: −9.7; 95% CI: −19.2, −0.3). By contrast, prenatal exposure to cannabis was associated with fewer internalizing behaviors (mean difference: −10.2; 95% CI: −20.3, −0.2) and fewer somatic complaints (mean difference: −5.2, 95% CI: −9.8, −0.6). Conclusions: Our study suggests that postnatal exposure to cannabis is associated with more behavioral and cognitive problems among 5-year-old children, independent of prenatal and postnatal exposure to tobacco. The potential risks of cannabis use (including smoking and vaping) during pregnancy and around young children should be more widely communicated to parents.