Sideband transitions are spectroscopically probed in a system consisting of a Cooper pair box strongly but nonresonantly coupled to a superconducting transmission line resonator. When the Cooper pair box is operated at the optimal charge bias point, the symmetry of the Hamiltonian requires a two-photon process to access sidebands. The observed large dispersive ac-Stark shifts in the sideband transitions induced by the strong nonresonant drives agree well with our theoretical predictions. Sideband transitions are important in realizing qubit-photon and qubit-qubit entanglement in the circuit quantum electrodynamics architecture for quantum information processing.