A Co$_2$FeSi (CFS) film with L2$_1$ structure was irradiated with different fluences of 30 keV Ga$^+$ ions. Structural modifications were subsequently studied using the longitudinal (LMOKE) and quadratic (QMOKE) magneto-optical Kerr effect. Both the coercivity and the LMOKE amplitude were found to show a similar behavior upon irradiation: they are nearly constant up to ion fluences of $≈6\times10^{15}$ ion/cm$^2$, while they decrease with further increasing fluences and finally vanish at a fluence of $≈9\times10^{16}$ ion/cm$^2$, when the sample becomes paramagnetic. However, contrary to this behavior, the QMOKE signal nearly vanishes even for the smallest applied fluence of $3\times10^{14}$ ion/cm$^2$. We attribute this reduction of the QMOKE signal to an irradiation-induced degeneration of second or higher order spin-orbit coupling, which already happens at small fluences of 30 keV Ga$^+$ ions. On the other hand, the reduction of coercivity and LMOKE signal with high ion fluences is probably caused by a reduction of the exchange interaction within the film material.