In order to explore the consequences of spin–orbit coupling on spin–phonon interactions in a set of chemically similar mixed metal oxides, we measured the infrared vibrational properties of Co4B2O9 (B = Nb, Ta) as a function of temperature and compared our findings with lattice dynamics calculations and several different models of spin–phonon coupling. Frequency vs temperature trends for the Co2+ shearing mode near 150 cm−1 reveal significant shifts across the magnetic ordering temperature that are especially large in relative terms. Bringing these results together and accounting for noncollinearity, we obtain spin–phonon coupling constants of −3.4 and −4.3 cm−1 for Co4Nb2O9 and the Ta analog, respectively. Analysis reveals that these coupling constants are derived from interlayer (rather than intralayer) exchange interactions and that the interlayer interactions contain competing antiferromagnetic and ferromagnetic contributions. At the same time, beyond-Heisenberg terms are minimized due to fortuitous symmetry considerations, different from most other 4d- and 5d-containing oxides. Comparison with other contemporary oxides shows that spin–phonon coupling in this family of materials is among the strongest ever reported, suggesting an origin for magnetoelectric coupling.