Water stress is the main abiotic factor that limits soybean grain yield. We investigated eight soybean cultivars under well-watered (WW) and terminal drought stress (TDS) conditions to determine the traits associated with water saving and the relationship between water use, root morphology, canopy architecture, flower and tagged-pod number, and yield performance. Under WW conditions, the average grain yield across the new soybean cultivars was significantly higher (18.7 g plant−1 vs. 15.1 g plant−1), but significantly less water was used (36 L plant−1 vs. 47 L plant−1) than in the old soybean cultivars. Under TDS, the four old soybean cultivars failed to produce a measurable grain yield, while the new soybean cultivars Zhonghuang 30 (ZH) and Jindou 19 (J19) produced a measurable grain yield. Water stress significantly reduced the flower numbers and tagged-pod numbers; the four new soybean cultivars on average had low flower and tagged-pod numbers under WW treatment, while they had low flower but high tagged-pod numbers under TDS conditions. ZH and JD exhibited a lower branch number and leaf area under both WW and TDS conditions. Water use during the flowering and podding periods was significantly positively correlated with the flower number and the tagged-pod number under both WW and TDS conditions. Thus, the small canopy size and low root length and root surface area contributed to a water-saving mechanism in the new soybean cultivars and improved the yield under drought conditions.