Understanding how individuals identify their relatives has implications for the evolution of social behaviour. Kinship cues might be based on familiarity, but in the face of paternity uncertainty and costly paternal investment, other mechanisms such as phenotypic matching may have evolved. In humans, paternal recognition of offspring and subsequent discriminative paternal investment have been linked to father–offspring facial phenotypic similarities. However, the extent to which paternity detection is impaired by environmentally induced facial information is unclear. We used 27 portraits of fathers and their adult sons to quantify the level of paternity detection according to experimental treatments that manipulate the location, type and quantity of visible facial information. We found that (i) the lower part of the face, that changes most with development, does not contain paternity cues, (ii) paternity can be detected even if relational information within the face is disrupted and (iii) the signal depends on the presence of specific information rather than their number. Taken together, the results support the view that environmental effects have little influence on the detection of paternity using facial similarities. This suggests that the cognitive dispositions enabling the facial detection of kinship relationships ignore genetic irrelevant facial information.