The scaling of Vickers hardness (Hv) in oxide glasses with varying network modifier/modifier ratio is manifested as either a positive or negative deviation from linearity with a maximum deviation at the ratio of about 1:1. In an earlier study [J. Kjeldsen et al., J. Non-Cryst. Solids 369, 61 (2013)], we observed a minimum of Hv in CaO/MgO sodium aluminosilicate glasses at CaO/MgO = 1:1 and postulated that this minimum is linked to a maximum in plastic flow. However, the origin of this link has not been experimentally verified. In this work, we attempt to do so by exploring the links among Hv, volume recovery ratio (VR) and plastic deformation volume (VP) under indentation, glass transition temperature (Tg), Young's modulus (E), and liquid fragility index (m) in CaO/MgO and CaO/Li2O sodium aluminosilicate glasses. We confirm the negative deviations from linearity and find that the maximum deviation (i.e., the so-called mixed modifier effect) of Hv, Tg, and m is at the modifier ratio of 1:1. These deviations increase in intensity as the total modifier concentration increases. We find a strong correlation between VP and Hv for the CaO/MgO series, implying that the minimum in Hv originates primarily from an increased shear flow in the mixed modifier glasses.