The host transmembrane protein SERINC5 is incorporated into retrovirus particles and inhibits HIV-1 infectivity. The lentiviral Nef protein counteracts SERINC5 by downregulating it from the cell surface and preventing its incorporation into virions. The ability of Nef to antagonize the host factor varies in magnitude between different HIV-1 isolates. After having identified a subtype H nef allele unable to promote HIV-1 infectivity in the presence of SERINC5, we investigated the molecular determinants responsible for the defective counteraction of the host factor. Chimeric molecules with a subtype C Nef highly active against SERINC5 were constructed to locate Nef residues crucial for the activity against SERINC5. An Asn at the base of the C-terminal loop of the defective nef allele was found in place of a highly conserved acidic residue (D/E 150). The conversion of Asn to Asp restored the ability of the defective Nef to downregulate SERINC5 and promote HIV-1 infectivity. The substitution was also found to be crucial for the ability of Nef to downregulate CD4, but not for Nef activities that do not rely on the internalization of receptors from the cell surface, suggesting a general implication in promoting clathrin-mediated endocytosis. Accordingly, bimolecular fluorescence complementation revealed that the conserved acidic residue contributes to the recruitment of AP2 by Nef. Altogether, our results confirm that Nef downregulates SERINC5 and CD4 by engaging a similar machinery and indicates that, in addition to the di-leucine motif, other residues in the C-terminal flexible loop are important for the ability of the protein to sustain clathrin-mediated endocytosis.