The matrix effect is one of the challenges to be overcome for a successful analysis of soil samples using X-ray fluorescence (XRF) sensors. This work aimed at evaluation of a simple modeling approach consisted of Compton normalization (CN) and multivariate regressions (e.g., multiple linear regressions (MLR) and partial least squares regression (PLSR)) to overcome the soil matrix effect, and subsequently improve the prediction accuracy of key soil fertility attributes. A portable XRF was used for analyzing 102 soil samples collected from two agricultural fields with contrasting soil matrices. Using the intensity of emission lines as input, preprocessing methods included with and without the CN. Univariate regression models for the prediction of clay, cation exchange capacity (CEC), and exchangeable (ex-) K and Ca were compared with the corresponding MLR models to assess matrix effect mitigation. The MLR and PLSR models improved the prediction results of the univariate models for both preprocessing methods, proving to be promising strategies for mitigating the matrix effect. In turn, the CN also mitigated part of the matrix effect for ex-K, ex-Ca, and CEC predictions, by improving the predictive performance of these elements when used in univariate and multivariate models. The CN has not improved the prediction accuracy of clay. The prediction performances obtained using MLR and PLSR were comparable for all evaluated attributes. The combined use of CN with multivariate regressions (MLR or PLSR) achieved excellent prediction results for CEC (R2 = 0.87), ex-K (R2 ≥ 0.94), and ex-Ca (R2 ≥ 0.96), whereas clay predictions were comparable with and without CN (0.89 ≤ R2 ≤ 0.92). We suggest using multivariate regressions (MLR or PLSR) combined with the CN to remove the soil matrix effects and consequently result in optimal prediction results of the studied key soil fertility attributes. The prediction performance observed for this solution showed comparable results to the approach based on the preprogrammed measurement package tested (Geo Exploration package, Bruker AXS, Madison, WI, USA).