Targeting C-X-C motif chemokine receptor 4 (CXCR4), as an oncogenic factor in Waldenström’s Macroglobulinemia (WM), has been considered a promising treatment strategy, and various CXCR4 antagonists, such as small molecules, peptides, and antibodies, are currently in preclinical and clinical phases. Recently, a 16-mer fragment of human serum albumin, the most abundant protein in plasma, has been identified as a potent inhibitor of CXCR4. Hence, it was given the name Endogenous Peptide Inhibitor of CXCR4 (EPI-X4) and demonstrated a less toxic alternative to the current therapies due to its improved selectivity profile. Further research has led to the discovery of optimized EPI-X4 derivatives, JM#21, and WSC02, as potent CXCR4 antagonists. Despite the ongoing research in this area, the molecular basis of how EPI-X4 and its derivatives bind to CXCR4 is still unexplored. In this study, three aspects were investigated using molecular dynamics (MD) simulations and binding energy calculations: (1) mapping the structure-function of the amino acid sequence composition of the peptides to determine the contribution of each amino acid towards the overall binding to CXCR4, as well as the key interaction peptide motifs; (2) the detailed binding mechanism of EPI-X4, JM#21, and WSC02 against CXCR4 at the molecular level; and (3) the impact of peptide binding on the conformational landscape of CXCR4. Per-residue energy decomposition analysis revealed that the inclusion of arginine residue in the peptide sequence has significantly improved the binding affinity towards CXCR4, particularly at positions Arg3 for EPI-X4 and WSC02 and Arg3 and Arg6 for JM#21. This may explain the finding that JM#21 exhibited the highest binding affinity. Additionally, MD simulations coupled with thermodynamic calculations provided a comprehensive picture of the “dynamic” interaction motif of the peptides with CXCR4. Separate MD simulations in a POPC bilayer were executed to determine its effect on the binding energy. This study provides molecular insights into the binding mechanism of the endogenous peptide EPI-X4 and its optimized derivatives with CXCR4, laying the basis for the rational design of optimized peptide derivatives against WM.