Food security is currently one of the most concerning problems in China. A traceability system is an effective method to improve the quality of food production. This system has been widely applied in several countries such as the US, Japan and the EU. However, in China, the creditability of traceability systems is not strong and many producers try to deceive the public by forging the data in such systems. In this paper, a video surveillance system-based traceability system is proposed which will significantly increase the forgery cost. In this system, subjects, such as vehicles or people, are firstly defined using a novel dynamic background model, then their trajectories are generated and connected using different cameras with a camera relation graph. The experimental results indicate that the proposed method can efficiently extract the object information from the video surveillance system and generate image-based traceability information to be used for further analysis.