In this paper, a novel three-dimensional (3D) indoor visible light positioning (VLP) algorithm using the Cayley–Menger determinant (CMD) with a cost function is proposed and experimentally tested to track a drone for industrial applications. The proposed algorithm uses optical received signal strength (RSS) for estimating the drone’s 3D position without prior knowledge of its height. This reduces the need for additional height sensors used in some 3D VLP systems. The performance of the proposed algorithm in terms of positioning error is also compared with a linear least squares (LLS) trilateration algorithm, with and without tilting of the receiver and with multipath reflections. The simulation results show that the proposed algorithm is more accurate and outperforms the LLS algorithm by a median improvement of 21% and is also more robust to the effect of tilting, as well as in the presence of multipath reflections. Furthermore, the proposed algorithm has been experimentally tested and compared with the LLS algorithm in a VLP test bed measuring 4 × 4 × 4.1 m 3 . The experimental results show that the median errors for LLS are 11.4 cm, while the median errors for CMD are 10.5 cm, which results in an error decrease of 8% when CMD with a cost function is used.