The organization of convection is ubiquitous, but its physical understanding remains limited. One particular type of organization is the spatial self-aggregation of convection, taking the form of cloud clusters, or tropical cyclones in the presence of rotation. We show that several physical processes can give rise to self-aggregation, and highlight the key features responsible for it using idealized simulations. Longwave radiative feedbacks yield a “radiative aggregation" . In that case, sufficient spatial variability of radiative cooling rates yields a low-level circulation, which induces the up-gradient energy transport and radiative-convective instability. Not only do vertically-integrated radiative budgets matter, but the vertical profile of cooling is also crucial. Convective aggregation is facilitated when downdrafts below clouds are weak (“moisture-memory aggregation" ), and this is sufficient to trigger aggregation in the absence of longwave radiative feedbacks. These results shed some light on the sensitivity of self-aggregation to various parameters, including resolution or domain size.