Photoreceptor cells (PRCs) are the light-detecting cells of the retina. Such cells can be non-invasively imaged using optical coherence tomography (OCT) which is used in clinical settings to diagnose and monitor ocular diseases. Here we present the largest genome-wide association study of PRC morphology to date utilising quantitative phenotypes extracted from OCT images within the UK Biobank. We discovered 111 loci associated with the thickness of one or more of the PRC layers, many of which had prior associations to ocular phenotypes and pathologies, and 27 with no prior associations. We further identified 10 genes associated with PRC thickness through gene burden testing using exome data. In both cases there was a significant enrichment for genes involved in rare eye pathologies, in particular retinitis pigmentosa. There was evidence for an interaction effect between common genetic variants, VSX2 involved in eye development and PRPH2 known to be involved in retinal dystrophies. We further identified a number of genetic variants with a differential effect across the macular spatial field. Our results suggest a continuum between common and rare variation which impacts retinal structure, sometimes leading to disease.