This paper proposes to employ the visual saliency for moving object detection via direct analysis from videos. Object saliency is represented by an information saliency map (ISM), which is calculated from spatio-temporal volumes. Both spatial and temporal saliencies are calculated and a dynamic fusion method developed for combination. We use dimensionality reduction and kernel density estimation to develop an efficient information theoretic based procedure for constructing the ISM. The ISM is then used for detecting foreground objects. Three publicly available visual surveillance databases, namely CAVIAR, PETS and OTCBVS-BENCH are selected for evaluation. Experimental results show that the proposed method is robust for both fast and slow moving object detection under illumination changes. The average detection rates are 95.42 % and 95.81 % while the false detection rates are 2.06 % and 2.40 % in CAVIAR (INRIA entrance hall and shopping center) dataset and OTCBVS-BENCH database, respectively. The average processing speed is 6.6 fps with frame resolution 320 × 240 in a typical Pentium IV computer. © 2009 Elsevier Ltd. All rights reserved.