Is it possible to meaningfully comprehend the diversity of the viral world? We propose that it is. This is based on the observation that, although there is immense genomic variation, every infective virion is restricted by strict constraints in structure space (i.e., there are a limited number of ways to fold a protein chain, and only a small subset of these have the potential to construct a virion, the hallmark of a virus). We have previously suggested the use of structure for the higher-order classification of viruses, where genomic similarities are no longer observable. Here, we summarize the arguments behind this proposal, describe the current status of structural work, highlighting its power to infer common ancestry, and discuss the limitations and obstacles ahead of us. We also reflect on the future opportunities for a more concerted effort to provide high-throughput methods to facilitate the large-scale sampling of the virosphere.