Author Posting. © The Author(s), 2012. This is the author's version of the work. It is posted here by permission of Elsevier B.V. for personal use, not for redistribution. The definitive version was published in Ocean Modelling 47 (2012): 65-95, doi:10.1016/j.ocemod.2012.01.003. ; The coupled ocean-atmosphere-wave-sediment transport modeling system (COAWST) enables simulations that integrate oceanic, atmospheric, wave and morphological processes in the coastal ocean. Within the modeling system, the three-dimensional ocean circulation module (ROMS) is coupled with the wave generation and propagation model (SWAN) to allow full integration of the effect of waves on circulation and vice versa. The existing wave-current coupling component utilizes a depth dependent radiation stress approach. In here we present a new approach that uses the vortex force formalism. The formulation adopted and the various parameterizations used in the model as well as their numerical implementation are presented in detail. The performance of the new system is examined through the presentation of four test cases. These include obliquely incident waves on a synthetic planar beach and a natural barred beach (DUCK’ 94); normal incident waves on a nearshore barred morphology with rip channels; and wave-induced mean flows outside the surf zone at the Martha’s Vineyard Coastal Observatory (MVCO). Model results from the planar beach case show good agreement with depth-averaged analytical solutions and with theoretical flow structures. Simulation results for the DUCK’ 94 experiment agree closely with measured profiles of cross-shore and longshore velocity data from Garcez-Faria et al. (1998, 2000). Diagnostic simulations showed that the nonlinear processes of wave roller generation and wave-induced mixing are important for the accurate simulation of surf zone flows. It is further recommended that a more realistic approach for determining the contribution of wave rollers and breaking induced turbulent mixing can be formulated using non-dimensional parameters which are functions of local wave parameters and the beach slope. Dominant terms in the cross-shore momentum balance are found to be the quasi-static pressure gradient and breaking acceleration. In the alongshore direction, bottom stress, breaking acceleration, horizontal advection and horizontal vortex forces dominate the momentum balance. The simulation results for the bar / rip channel morphology case clearly show the ability of the modeling system to reproduce horizontal and vertical circulation patterns similar to those found in laboratory studies and to numerical simulations using the radiation stress representation. The vortex force term is found to be more important at locations where strong flow vorticity interacts with the wave-induced Stokes flow field. Outside the surf zone, the three-dimensional model simulations of wave-induced flows for non- breaking waves closely agree with flow observations from MVCO, with the vertical structure of the simulated flow varying as a function of the vertical viscosity as demonstrated by Lentz et al. (2008). ; The first two authors were supported by a NOAA/IOOS Grant (Integration of Coastal Observations and Assets in the Carolinas in Support of Regional Coastal Ocean Observation System Development in the Southeast Atlantic) and a cooperative agreement between U.S. Geological Survey and University of South Carolina as part of the Carolinas Coastal Change Processes Project. Also G. Voulgaris was partially supported by the National Science Foundation (Awards: OCE-0451989 and OCE-0535893).