Abstract We perform a systematic simulation study of energetic passing particle-driven instabilities in KSTAR using the kinetic-MHD hybrid code M3D-K. Linear simulation results show that the observed n = 1 mode in the early phase of the discharge is the low-frequency fishbone driven by energetic passing beam ions. The mode frequency computed is in a good agreement with the experimental measurement. Nonlinear simulations show that the frequency of the n = 1 mode jumps up to a higher value corresponding to the β-induced Alfv ́en eigenmode (BAE). In the later phase of the discharge, the simulated n = 5 mode is identified as a BAE in its linear phase. In the nonlinear phase, the n = 5 mode exhibits a similar frequency jump to a higher value of an energetic particle mode (EPM) after mode saturation. Analysis of perturbed beam ion distributions in phase space shows that these new modes in nonlinear stages are driven by new resonances due to nonlinearly evolved beam ion distributions. Further simulations of a beam beta scan for the n = 5 mode show that the frequency jump disappears for a sufficiently small beam beta or beam ion drive. This result may explain the non-existence of frequency jump in the experiment. Finally, the impact of toroidal rotation on mode characteristics is investigated, showing that it has a marginal influence on energetic particle driven modes.