Device-to-device (D2D) communication has attracted many researchers, cellular operators, and equipment makers as mobile traffic and bandwidth demands have increased. It supports direct communication within devices with no need for any intermediate node and, therefore, offers advantage in 5G network while providing wide cell coverage range and frequency reuse. However, establishing acceptable and secure mechanism for D2D communication which ensures confidentiality, integrity, and availability is an issue encountered in this situation. Furthermore, in a resource-constrained IoT environment, these security challenges are more critical and difficult to mitigate, especially during emergence of IoT with 5G network application scenarios. To address these issues, this paper proposed a security mechanism in 5G network for D2D wireless communication dependent on lightweight modified elliptic curve cryptography (LMECC). The proposed scheme follows a proactive routing protocol to discover services, managing link setup, and for data transfer with the aim to reduce communication overhead during user authentication. The proposed approach has been compared against Diffie–Hellman (DH) and ElGamal (ELG) schemes to evaluate the protocol overhead and security enhancement at network edge. Results proved the outstanding performance of the proposed LMECC for strengthening data secrecy with approximate 13% and 22.5% lower overhead than DH and ELG schemes.