We demonstrate here the development of a non-invasive optical forward-scattering system, called 'scatterometer' for rapid identification of bacterial colonies. The system is based on the concept that variations in refractive indices and size, relative to the arrangement of cells in bacterial colonies growing on a semi-solid agar surface will generate different forward-scattering patterns. A 1.2-1.5mm colony size for a 1mm laser beam and brain heart infusion agar as substrate were used as fixed variables. The current study is focused on exploring identification of Listeria monocytogenes and other Listeria species exploiting the known differences in their phenotypic characters. Using diffraction theory, we could model the scattering patterns and explain the appearance of radial spokes and the rings seen in the scattering images of L. monocytogenes. Further, we have also demonstrated development of a suitable software for the extraction of the features (scalar values) calculated from images of the scattering patterns using Zernike moment invariants and principal component analysis and were grouped using K-means clustering. We achieved 91-100% accuracy in detecting different species. It was also observed that substrate variations affect the scattering patterns of Listeria. Finally, a database was constructed based on the scattering patterns from 108 different strains belonging to six species of Listeria. The overall system proved to be simple, non-invasive and virtually reagent-less and has the potential for automated user-friendly application for detection and differentiation of L. monocytogenes and other Listeria species colonies grown on agar plates within 5-10 min analysis time.