Schizophrenia spectrum and bipolar disorder are characterized by high levels of childhood trauma as well as of cognitive dysfunction. Our aim is to investigate the association between these two factors in the largest study in the literature so far. A total of 406 patients with schizophrenia spectrum- or bipolar disorders were recruited from a catchment area based organization in Oslo, Norway. Information about early life stress was obtained using the Childhood Trauma Questionnaire (CTQ). Cognitive function was assessed through a comprehensive and standardized neuropsychological test battery. Physical abuse, sexual abuse and physical neglect were significantly associated with reduced scores on working memory and executive function scales (p=0.04 to p<0.001), and verbal and performance tasks from the Wechsler Abbreviated Scale of Intelligence (WASI) (p=0.059 to p<0.001). When verbal and performance tasks from the WASI were added into a multivariate regression model, the association between CTQ and the specific cognitive domains decreased, and only WASI scores remained statistically significant. Our results indicate that childhood trauma is associated with a reduction in cognitive function across cognitive domains in patients with schizophrenia spectrum- and bipolar disorders, in particular working memory and executive function as well as general cognition. Moreover, these dysfunctions seem to be driven by underlying deficits in general cognitive tasks as measured by the WASI.