It has been proposed that valence-band holes can form robust spin qubits1, 2, 3, 4 owing to their weaker hyperfine coupling compared with electrons5, 6. However, it was demonstrated recently7, 8, 9, 10, 11 that the hole hyperfine interaction is not negligible, although a consistent picture of the mechanism controlling its magnitude is still lacking. Here we address this problem by measuring the hole hyperfine constant independently for each chemical element in InGaAs/GaAs, InP/GaInP and GaAs/AlGaAs quantum dots. Contrary to existing models10, 11 we find that the hole hyperfine constant has opposite signs for cations and anions and ranges from −15% to +15% relative to that for electrons. We attribute such changes to the competing positive contributions of p-symmetry atomic orbitals and the negative contributions of d-orbitals. These findings yield information on the orbital composition of the valence band12 and enable a fundamentally new approach for verification of computed Bloch wavefunctions in semiconductor nanostructures13. Furthermore, we show that the contribution of cationic d-orbitals leads to a new mechanism of hole spin decoherence.