Ecological networks depict the interactions between species, mainly based on observations in the field. The information contained in such interaction matrices depends on the sampling design, and typically, compounds preferences (specialization) and abundances (activity). Null models are the primary vehicles to disentangle the effects of specialization from those of sampling and abundance, but they ignore the feedback of network structure on abundances. Hence, network structure, as exemplified here by modularity, is difficult to link to specific causes. Indeed, various processes lead to modularity and to specific interaction patterns more generally. Inferring (co)evolutionary dynamics is even more challenging, as competition and trait matching yield identical patterns of interactions. A satisfactory resolution of the underlying factors determining network structure will require substantial additional information, not only on independently assessed abundances, but also on traits, and ideally on fitness consequences as measured in experimental setups.