The approaches based on background difference are the most used with fixed cameras to perform motion detection, because of the high quality of the segmentation achieved. However, real time requirements prevent most of the al- gorithms proposed in literature to exploit the background difference with Pan-Tilt-Zoom (PTZ) cameras in real world applications. Nevertheless, using color information to de- tect motion yields a sensible improvement in terms of accu- racy of segmented masks, by helping reducing camouflage and detecting shadows. To our knowledge, the algorithm we have conceived is the first to exploit color information and perform real time alignment and background differ- ence based on background mosaicing, with PTZ cameras. In addition, no prior information regarding scene or cam- era parameters have been used for either spatial or tonal alignment. Accurate experiments performed on indoor and outdoor sequences allow to assess both quality and perfor- mance of the method we devised.