The chemical evolution of galaxies is investigated within the framework of the star formation rate (SFR) dependent integrated galactic initial mass function (IGIMF). We study how the global chemical evolution of a galaxy and in particular how [alpha/Fe] abundance ratios are affected by the predicted steepening of the IGIMF with decreasing SFR. We use analytical and semi-analytical calculations to evaluate the mass-weighted and luminosity-weighted [alpha/Fe] ratios in early-type galaxies of different masses. The models with the variable IGIMF produce a [alpha/Fe] vs. velocity dispersion relation which has the same slope as the observations of massive galaxies, irrespective of the model parameters, provided that the star formation duration inversely correlates with the mass of the galaxy (downsizing). These models also produce steeper [alpha/Fe] vs. sigma relations in low-mass early-type galaxies and this trend is consistent with the observations. Constant IMF models are able to reproduce the [alpha/Fe] ratios in large elliptical galaxies as well, but they do not predict this change of slope for small galaxies. In order to obtain the best fit between our results and the observations, the downsizing effect (i.e. the shorter duration of the star formation in larger galaxies) must be milder than previously thought.