Cortical inhibitory neurons contact each other to form a network of inhibitory synaptic connections. Our knowledge of the connectivity pattern underlying this inhibitory network is, however, still incomplete. Here we describe a simple and complementary interaction scheme between three large, molecularly distinct interneuron populations in mouse visual cortex: parvalbumin-expressing interneurons strongly inhibit one another but provide little inhibition to other populations. In contrast, somatostatin-expressing interneurons avoid inhibiting one another yet strongly inhibit all other populations. Finally, vasoactive intestinal peptide-expressing interneurons preferentially inhibit somatostatin-expressing interneurons. This scheme occurs in supragranular and infragranular layers, suggesting that inhibitory networks operate similarly at the input and output of the visual cortex. Thus, as the specificity of connections between excitatory neurons forms the basis for the cortical canonical circuit, the scheme described here outlines a standard connectivity pattern among cortical inhibitory neurons.