Extensive surveys of star-forming regions with Spitzer have revealed populations of disk-bearing young stellar objects. These have provided crucial constraints, such as the timescale of dispersal of protoplanetary disks, obtained by carefully combining infrared data with spectroscopic or X-ray data. While observations in various regions agree with the general trend of decreasing disk fraction with age, the Lupus V and VI regions appeared to have been at odds, having an extremely low disk fraction. Here we show, using the recent Gaia data release 2 (DR2), that these extremely low disk fractions are actually due to a very high contamination by background giants. Out of the 83 candidate young stellar objects (YSOs) in these clouds observed by Gaia, only five have distances of ~150 pc, similar to YSOs in the other Lupus clouds, and have similar proper motions to other members in this star-forming complex. Of these five targets, four have optically thick (Class II) disks. On the one hand, this result resolves the conundrum of the puzzling low disk fraction in these clouds, while, on the other hand, it further clarifies the need to confirm the Spitzer selected diskless population with other tracers, especially in regions at low galactic latitude like Lupus V and VI. The use of Gaia astrometry is now an independent and reliable way to further assess the membership of candidate YSOs in these, and potentially other, star-forming regions.