Sense of agency refers to the feeling that one’s self-generated action caused an external environment event. In a previous study, we suggested that the supplementary motor area (SMA), in its anterior portion (pre-SMA), is a key structure for attributing the sense of agency for the visual consequences of self-generated movements. However, real-life actions can lead to outcomes in different sensory modalities, raising the question of whether SMA represents a supra-modal hub for the sense of agency. Here, we compared the agency experience for visual and auditory outcomes by taking advantage of the intentional binding effect (IB). We observed discrete time-windows for the agency manifestation across different sensory modalities: While there was an IB at 200 ms delay between the action and the visual outcome, a time compression was observed when the auditory outcome followed the action by 400 ms. The magnitude of the IB was mirrored by meaningful brain activity in the pre-SMA but only at the specific delay when a sizeable IB was seen. We conclude that attributing consequences of self-generated movements to one’s action is based on similar mechanisms across sensory modalities and that those mechanisms are related to the functioning of the motor system.