The Recurrent Temporal Restricted Boltzmann Machine is a promising probabilistic model for processing temporal data. It has been shown to learn physical dynamics from videos (e.g. bouncing balls), but its ability to process sequential data has not been tested on symbolic tasks. Here we assess its capabilities on learning sequences of letters correspond-ing to English words. It emerged that the model is able to extract local transition rules between items of a sequence (i.e. English graphotactic rules), but it does not seem to be suited to encode a whole word.