Multi sensor fusion has been widely used in recognition problems. Most existing works highly depend on the calibration between different sensors, but less on modeling and reasoning of the co-incidence of multiple hints. In this paper, we propose a generic framework for recognition and clustering problem using a non-parametric Dirichlet hierarchical model, named DP-Fusion. It enables online labeling, clustering and recognition of sequential data simultaneously, while considering multiple types of sensor readings. The algorithm is data-driven, which does not depend on priorknowledge of the data structure. The results show the feasibility and reliability against noise data.