Based on a newly constructed Mg-Cu-Ni n-body potential, atomistic simulations revealed the underlying mechanism of metallic glass formation is the crystalline lattice collapsing while solute concentration exceeding a critical value, and predicted a quadrilateral region in the composition triangle, energetically favoring the formation of ternary Mg-Cu-Ni metallic glasses. Moreover, an optimized stoichiometry area around Mg60Cu17Ni23 was further located, at which the driving force for transforming the crystalline solid solution into a disordered state, i.e., the glassy phase reaches its maximum. Furthermore, by characterizing the local environments, the frustration of crystallization favorable short-range orders was revealed to be correlated with the optimum glass forming ability (GFA) in Mg-Cu-Ni system, interpreting the structural orgin of GFA and lending further support to the prediction results.