Genetic risk for schizophrenia has a negative impact on memory and other cognitive abilities in unaffected individuals, and it was recently shown that this effect is specific to males. Using functional MRI, we investigated the effect of a polygenic risk score (PRS) for schizophrenia on brain activation during working memory and episodic memory in 351 unaffected participants (167 males and 184 females, 25–95 years), and specifically tested if any effect of PRS on brain activation is sex-specific. Schizophrenia PRS was significantly associated with decreased brain activation in the left dorsolateral prefrontal cortex (DLPFC) during working-memory manipulation and in the bilateral superior parietal lobule (SPL) during episodic-memory encoding and retrieval. A significant interaction effect between sex and PRS was seen in the bilateral SPL during episodic-memory encoding and retrieval, and sex-stratified analyses showed that the effect of PRS on SPL activation was male-specific. These results confirm previous findings of DLPFC inefficiency in schizophrenia, and highlight the SPL as another important genetic intermediate phenotype of the disease. The observed sex differences suggest that the previously shown male-specific effect of schizophrenia PRS on cognition translates into an additional corresponding effect on brain functioning.