Context. Radial velocities (RVs) measured from high-resolution stellar spectra are routinely used to detect and characterise orbiting exoplanet companions. The different lines present in stellar spectra are created by several species, which are non-uniformly affected by stellar variability features such as spots or faculae. Stellar variability distorts the shape of the spectral absorption lines from which precise RVs are measured, posing one of the main problems in the study of exoplanets. Aims. In this work we aim to study how the spectral lines present in M dwarfs are independently impacted by stellar activity. Methods. We used CARMENES optical spectra of six active early- and mid-type M dwarfs to compute line-by-line RVs and study their correlation with several well-studied proxies of stellar activity. Results. We are able to classify spectral lines based on their sensitivity to activity in five M dwarfs displaying high levels of stellar activity. We further used this line classification to compute RVs with activity-sensitive lines and less sensitive lines, enhancing or mitigating stellar activity effects in the RV time series. For specific sets of the least activity-sensitive lines, the RV scatter decreases by ~2 to 5 times the initial one, depending on the star. Finally, we compare these lines in the different stars analysed, finding the sensitivity to activity to vary from star to star. Conclusions. Despite the high density of lines and blends present in M dwarf stellar spectra, we find that a line-by-line approach is able to deliver precise RVs. Line-by-line RVs are also sensitive to stellar activity effects, and they allow for an accurate selection of activity-insensitive lines to mitigate activity effects in RV. However, we find stellar activity effects to vary in the same insensitive lines from star to star.