Cadmium contamination is a severe threat to food safety. Therefore, the development of sensitive and selective cadmium detection strategies is urgently required. The elimination of background autofluorescence generated from the food matrix is critical to the optical assay for cadmium detection. Herein, a time-resolved phosphorescence sensor based on an “on-off” strategy was developed for cadmium determination in food samples. The phosphorescence nanoparticles were used as a luminous material to minimize the interference of background autofluorescence. The cadmium-binding aptamer was immobilized onto the magnetic beads and combined with a black hole quencher 1 (BHQ1) with complementary DNA as the target recognition element. With the presence of cadmium, the cadmium-binding aptamer bound to cadmium specifically and resulted in the release of BHQ1. The free BHQ1 remained in the solution after magnetic separation and quenched the phosphorescence. The phosphorescence intensity was negatively related to the concentration of cadmium. Under optimal conditions, the time-resolved phosphorescence sensor showed a linear response to cadmium concentration within a range from 0.05 to 5 ng mL−1 and with a detection limit of 0.04 ng mL−1. This “on-off” time-resolved phosphorescence sensor was successfully applied for cadmium detection in spring water and clam samples, which provided a rapid and straightforward method.