Aflatoxin M1 (AFM1) is one of the most widespread aflatoxins that can be present in the milk of lactating mammals. It can cause carcinogenicity, mutagenesis, teratogenesis, genotoxicity and immunosuppression. The WHO recommends reducing the AFM1 concentration in food products, so the European Commission has set a maximum allowable limit of 0.05 µg L−1 in milk and its products. Thus, there is a need to develop new methodologies to satisfy the demand for reliable, cost-effective, robust and sensitive AFM1 routine controls. In the present work, a competitive phosphorescent immunosensor for AFM1 quantification in milk, based on antibody–antigen recognition and Mn:ZnS quantum dots (d-QDs) as photoluminescent labels, has been developed. Two different assay strategies based on the use of d-QDs as labels of secondary antibodies (direct assay), or of a derivative species of the antigen AFM1-Bovine Serum Albumin (indirect assay) were compared in terms of analytical performance for AFM1 quantification. The best analytical results were obtained with the immunoassay format that uses d-QDs as tags of secondary antibodies (direct assay), and said design was finally selected. The selected immunosensor provided a detection limit for AFM1 quantification of only 0.002 µg L−1, which greatly satisfied the maximum tolerable limit of AFM1 in milk of 0.05 µg L−1. The accuracy, calculated as recovery of AFM1 in fortified skimmed milk samples, ranged from 81 to 90%, with relative standard deviations from 3% to 14%. These results bring to light the good performance of such phosphorescent biosensors as simple and fast alternatives to conventional chromatographic analytical methods.