Polygoni Vivipari Rhizoma (PVR), the dried root of Polygonum viviparum, has been used as herbal medicine in China for a long time. In the present study, a new method based on multi-step matrix solid-phase dispersion (MSPD), ultrafiltration and high performance liquid chromatography (HPLC) for screening alpha-glucosidase inhibitors (AGIs) from PVR was proposed. First, three different PVR extractions were prepared by multi-step MSPD with 15% methanol, 60% methanol and 100% methanol. Second, the alpha-glucosidase inhibition tests for the three extracts were carried out, and the 60% methanol extraction showed the best activity. Then, the AGIs screening experiment was performed with ultrafiltration and HPLC analysis using the 60% methanol extraction. Seven binding components (quercetin−3−O−vicianoside, quercetin 3−O−neohesperidoside, rutin, hyperoside, quercetin 3−O−glucuronide, luteolin−7−O−neohesperidoside, kaempferol 3−glucuronide) were found. These seven components were further validated as the AGIs by molecular docking analysis. The developed method was a rapid and efficient tool for screening AGIs from PVR, which provided scientific data for the bioactive components study of PVR.