Structural brain magnetic resonance imaging (MRI) traits share part of their genetic variance with cognitive traits. Here, we use genetic association results from large meta-analytic studies of genome-wide association (GWA) for brain infarcts (BI), white matter hyperintensities, intracranial, hippocampal, and total brain volumes to estimate polygenic scores for these traits in three Scottish samples: Generation Scotland: Scottish Family Health Study (GS:SFHS), and the Lothian Birth Cohorts of 1936 (LBC1936) and 1921 (LBC1921). These five brain MRI trait polygenic scores were then used to: (1) predict corresponding MRI traits in the LBC1936 (numbers ranged 573 to 630 across traits), and (2) predict cognitive traits in all three cohorts (in 8,115–8,250 persons). In the LBC1936, all MRI phenotypic traits were correlated with at least one cognitive measure, and polygenic prediction of MRI traits was observed for intracranial volume. Meta-analysis of the correlations between MRI polygenic scores and cognitive traits revealed a significant negative correlation (maximal r = 0.08) between the HV polygenic score and measures of global cognitive ability collected in childhood and in old age in the Lothian Birth Cohorts. The lack of association to a related general cognitive measure when including the GS:SFHS points to either type 1 error or the importance of using prediction samples that closely match the demographics of the GWA samples from which prediction is based. Ideally, these analyses should be repeated in larger samples with data on both MRI and cognition, and using MRI GWA results from even larger meta-analysis studies.