This paper evaluates different Restricted Boltzmann Machines models in unsupervised, semi-supervised and supervised frameworks using information from human actions. After feeding these multilayer models with low level features, we infer high-level discriminating features that highly improve the classification performance. This approach eliminates the difficult process of selecting good mid-level feature descriptors, changing the feature selection and extraction process by a learning stage. Two main contributions are presented. First, a new sequence-descriptor from accumulated histograms of optical flow (aHOF) is presented. Second, comparative results using unsupervised, supervised and semi-supervised classification experiments are shown. The results show that the RBM architectures provide very good results in our classification task and present very good properties for semi-supervised learning.