We study the effect of valence band spin–orbit interactions (SOI) on the acoustic phonon-assisted spin relaxation of holes confined in quantum dots (QDs). Heavy hole–light hole (hh–lh) mixing and all the spin–orbit terms arising from zinc-blende bulk inversion asymmetry (BIA) are considered on equal footing in a fully three-dimensional Hamiltonian. We show that hh–lh mixing and BIA have comparable contributions to the hole spin relaxation in self-assembled QDs, but BIA becomes dominant in gated QDs. Simultaneously accounting for both mechanisms is necessary for quantitatively correct results in quasi-two-dimensional QDs. The dependence of the hole spin relaxation on the QD geometry and spin splitting energy is drastically different from that of electrons, with a non-monotonic behavior which results from the interplay between SOI terms. Our results reconcile contradictory predictions of previous theoretical works and are consistent with experiments.