High-energy heavy ion irradiation can produce permanent damage in the target material if the density of deposited energy surpasses a material-dependent threshold value. It is known that this threshold can be lowered in the vicinity of the surface or in the presence of defects. In the present study, we established threshold values for Al2O3, MgO and CaF2 under the above-mentioned conditions, and found those values to be much lower than expected. By means of atomic force microscopy and Rutherford backscattering spectrometry in channelling mode, we present evidence that ion beams with values of 3 MeV O and 5 MeV Si, despite the low density of deposited energy along the ion trajectory, can modify the structure of investigated materials. The obtained results should be relevant for radiation hardness studies because, during high-energy ion irradiation, unexpected damage build-up can occur under similar conditions.