The baculum is an enigmatic bone within the mammalian glans penis, and the driving forces behind its often bizarre shape have captivated evolutionary biologists for over a century. Hypotheses for the function of the baculum include aiding in intromission, stimulating females and assisting with prolonged mating. Previous attempts to test these hypotheses have focused on the gross size of the baculum and have failed to reach a consensus. We conducted three-dimensional imaging and apply a new method to quantify three-dimensional shape complexity in the carnivoran baculum. We show that socially monogamous species are evolving towards complex-shaped bacula, whereas group-living species are evolving towards simple bacula. Overall three-dimensional baculum shape complexity is not related to relative testes mass, but tip complexity is higher in induced ovulators and species engaging in prolonged copulation. Our study provides evidence of postcopulatory sexual selection pressures driving three-dimensional shape complexity in the carnivore baculum.