Adaptor protein complex 5 (AP-5) and its partners, SPG11 and SPG15, are recruited onto late endosomes and lysosomes. Here we show that recruitment of AP-5/SPG11/SPG15 is enhanced in starved cells and occurs by coincidence detection, requiring both phosphatidylinositol 3-phosphate (PI3P) and Rag GTPases. PI3P binding is via the SPG15 FYVE domain, which, on its own, localizes to early endosomes. GDP-locked RagC promotes recruitment of AP-5/SPG11/SPG15, while GTP-locked RagA prevents its recruitment. Our results uncover an interplay between AP-5/SPG11/SPG15 and the mTORC1 pathway and help to explain the phenotype of AP-5/SPG11/SPG15 deficiency in patients, including the defect in autophagic lysosome reformation.