γ-triclinic and α-monoclinic polymorphs of CoV2O6 are two of the few known transition-metal ion-based materials that display stepped 13 magnetization plateaus at low temperatures. Neutron diffraction [M. Markkula, Phys. Rev. B 86, 134401 (2012)]PRBMDO1098-012110.1103/PhysRevB.86.134401, x-ray dichroism [N. Hollmann, Phys. Rev. B 89, 201101(R) (2014)]PRBMDO1098-012110.1103/PhysRevB.89.201101, and dielectric measurements [J. Singh, J. Mater. Chem. 22, 6436 (2012)]JMACEP0959-942810.1039/c2jm16290c have shown a coupling between orbital, magnetic, and structural orders in CoV2O6. We apply neutron inelastic scattering to investigate this coupling by measuring the spin-orbit transitions in both α and γ polymorphs. We find the spin exchange and anisotropy in monoclinic α-CoV2O6 to be weak in comparison with the spin-orbit coupling λ and estimate an upper limit of J/λ ∼0.05. However, the spin exchange is larger in the triclinic polymorph and we suggest the excitations are predominately two dimensional. The local compression of the octahedra surrounding the Co2+ ion results in a direct coupling between higher-energy orbital levels, the magnetic ground state, and elastic strain. CoV2O6 is therefore an example where the local distortion along with the spin-orbit coupling provides a means of intertwining structural and magnetic properties. We finish the paper by investigating the low-energy magnetic fluctuations within the ground-state doublet and report a magnetic excitation that is independent of the local crystalline electric field. We characterize the temperature and momentum dependence of these excitations and discuss possible connections to the magnetization plateaus.