Transcription factors often activate and repress different target genes in the same cell. How activation and repression are encoded by different arrangements of transcription factor binding sites in cis-regulatory elements is poorly understood. We investigated how sites for the transcription factor CRX encode both activation and repression in photoreceptors by assaying thousands of genomic and synthetic cis-regulatory elements in wild-type and Crx−/− retinas. We found that sequences with high affinity for CRX repress transcription, whereas sequences with lower affinity activate. This rule is modified by a cooperative interaction between CRX sites and sites for the transcription factor NRL, which overrides the repressive effect of high affinity for CRX. Our results show how simple rearrangements of transcription factor binding sites encode qualitatively different responses to a single transcription factor and explain how CRX plays multiple cis-regulatory roles in the same cell.