How can a human collective coordinate, for example to move a banquet table, when each person is influenced by the inertia of others who may be inferior at the task? We hypothesized that large groups cannot coordinate through touch alone, accruing to a zero-sum scenario where individuals inferior at the task hinder superior ones. We tested this hypothesis by examining how dyads, triads and tetrads, whose right hands were physically coupled together, followed a common moving target. Surprisingly, superior individuals followed the target accurately even when coupled to an inferior group, and the interaction benefits increased with the group size. A computational model shows that these benefits arose as each individual uses their respective interaction force to infer the collective’s target and enhance their movement planning, which permitted coordination in seconds independent of the collective’s size. By estimating the collective’s movement goal, its individuals make physical interaction beneficial, swift and scalable.