This paper introduces the covariance matrix of visually salient image features as a compact and robust descriptor for near duplicate image and video copy detection. We make two novel contributions. We first present a fast method for computing information theoretic based visual saliency maps using a data independent fast transform to replace the conventional data dependent computationally demanding transforms. We then introduce salient covariance (SCOV) — the covariance matrix of various image features within the visually salient regions and use SCOV for near duplicate image and video copy detection. We present experimental results to show that our new fast visual saliency computation technique improves efficiency without compromising performances. We demonstrate that SCOV is a very compact and robust feature for near duplicate image and video copy detection. Compared to popular features such as GIST, SCOV is not only more robust against various manipulations but also can be over 20 times more compact whilst achieving the same or better performances.