Aims. In this study, we introduce a general paradigm for generating independent and well-balanced training, validation, and test sets for use in supervised machine and deep learning flare forecasting, to determine the extent to which video-based deep learning can predict solar flares. Methods. We use this implementation paradigm in the case of a deep neural network, which takes videos of magnetograms recorded by the Helioseismic and Magnetic Imager onboard the Solar Dynamics Observatory (SDO/HMI) as input. Results. The way the training and validation sets are prepared for network optimization has a significant impact on the prediction performances. Furthermore, deep learning is able to realize flare video classification with prediction performances that are in line with those obtained by machine learning approaches that require an a priori extraction of features from the HMI magnetograms. Conclusions. To our knowledge, this is the first time that the solar flare forecasting problem is addressed by means of a deep neural network for video classification, which does not require any a priori extraction of features from the HMI magnetograms.