Conventional weed control practices have generated serious issues related to the environment and human health. Therefore, there is a demand for the development of alternative techniques for sustainable agriculture. The present study performed a large-scale screening of allelopathic bacteria from the rhizosphere of weeds and wheat to obtain biological weed control inoculants in the cultivation of wheat. Initially, around 400 strains of rhizobacteria were isolated from the rhizosphere of weeds as well as wheat that grows in areas of chronic weed invasions. A series of the screen was performed on these strains, including the release of phytotoxic metabolites, growth inhibition of sensitive Escherichia coli, growth inhibition of indicator plant of lettuce, agar bioassays on five weeds, and agar bioassay on wheat. Firstly, 22.6% (89 strains) of the total strains were cyanogenic, and among the cyanogenic strains, 21.3% (19 strains) were inhibitory to the growth of sensitive E. coli. Then, these 19 strains were tested using lettuce seedling bioassay to show that eight strains suppressed, nine strains promoted, and two strains remained ineffective on the growth. These 19 strains were further applied to weeds and wheat on agar bioassays. The results indicated that dry matter of broad-leaved dock, wild oat, little seed canary grass, and common lambs’ quarter were reduced by eight strains (23.1–68.1%), seven strains (38.5–80.2%), eight strains (16.5–69.4%), and three strains (27.5–50.0%), respectively. Five strains suppressed the growth of wheat, nine strains increased its dry matter (12.8–47.9%), and five remained ineffective. Altogether, the strains that selectively inhibit weeds, while retaining normal growth of wheat, can offer good opportunities for the development of biological weed control in the cultivation of wheat.