The quantum Rabi model describes the ultrastrong interaction of a two-level atom coupled to a single quantized bosonic mode. As compared to the Jaynes-Cummings model, in the Rabi model the absorption and emission processes do not need to satisfy energy conservation and the usual rotating wave approximation (RWA) breaks down. As a result, the atom-field dynamics in the Hilbert space splits into two independent parity chains, exhibiting a collapse-revival pattern and exact periodic dynamics in the limit of degenerate atomic levels. Here we introduce a mixed Rabi Jaynes-Cummings model by considering a three-level atom interacting with two quantized bosonic fields, in which the RWA is made for one transition (with a weak atom-field coupling) but not for the other one (with an ultrastrong atom-field coupling). As a result, we show that the field in the weak coupled atomic transition can be used as a tool to control the atom-field dynamics of the other (strong coupled) transition, thus realizing an effective two-level quantum Rabi model with a controllable field. In particular, a periodic temporal dynamics of the atom-field state can be realized by appropriate tuning of the weak control field, even for non-degenerate atomic levels. A photonic simulator of the mixed Rabi Jaynes-Cummings model, based on light transport in evanescently-coupled optical waveguide lattices, is also briefly discussed.