This article highlights challenges associated with securing visible light communication (VLC) systems by using physical layer security (PLS) techniques. Motivated by the achievements in PLS studies for radio frequency (RF) communication, many PLS techniques for VLC systems were also rigorously investigated by tailoring the RF techniques to the VLC environment. However, careful consideration of the inherent differences between RF and VLC systems is still needed. By disregarding these differences, an eavesdropper could be given an opportunity to wiretap the VLC systems, even when PLS techniques are employed to protect them. Crucially, the fact that it is often not possible to know the number and locations of eavesdroppers in real VLC systems may allow eavesdroppers to devise various cooperative eavesdropping methods. By examining a few examples of the possible eavesdropper threats that can occur in VLC systems, this article offers novel insights into the vulnerabilities of state-of-the-art PLS schemes for VLC systems. Although the focus of the paper is mostly on these weaknesses, some potential solutions are also briefly proposed with a view to stimulating discourse in the community.