This paper studies various mechanisms to enhance the coherent Smith-Purcell radiation (SPR) at terahertz frequencies using particle-in-cell (PIC) simulations. A simple analytical dispersion relation is derived to predict the frequency of the evanescent surface wave of the open grating, which is excited by a backward-wave oscillation mechanism and emits radiation at harmonics of this frequency, and the cavity mode frequencies of the Orotron-like device, all in excellent agreement with the simulation data. The grating parameters are optimized to minimize the starting current of a continuous electron beam for exciting coherent SPR. It is demonstrated that a prebunched beam can be used to selectively excite coherent SPR at desired SPR angle and frequencies that are higher harmonics of the beam bunching frequency, where the starting current requirement is eliminated. Our PIC simulations show the dynamics of modes development and competition for the Orotron-like open cavity structure, which owns a lower starting current than the surface mode process. This oscillation mode dominates at steady state and enhances the radiated power by more than 2 orders of magnitude.