Abstract Interplanetary coronal mass ejections (ICMEs) are known to modify the structure of the solar wind as well as interact with the space environment of planetary systems. Their large magnetic structures have been shown to interact with galactic cosmic rays (GCRs), leading to the Forbush decrease (FD) phenomenon. We revisit in the present article the 17 yr of Advanced Composition Explorer spacecraft ICME detection along with two neutron monitors (McMurdo and Oulu) with a superposed epoch analysis to further analyze the role of the magnetic ejecta in driving FDs. We investigate in the following the role of the sheath and the magnetic ejecta in driving FDs, and we further show that for ICMEs without a sheath, a magnetic ejecta only is able to drive significant FDs of comparable intensities. Furthermore, a comparison of samples with and without a sheath with similar speed profiles enable us to show that the magnetic field intensity, rather than its fluctuations, is the main driver for the FD. Finally, the recovery phase of the FD for isolated magnetic ejecta shows an anisotropy in the level of the GCRs. We relate this finding at 1 au to the gradient of the GCR flux found at different heliospheric distances from several interplanetary missions.