Cardiac arrhythmias pose a significant danger to human life; therefore, it is of utmost importance to be able to efficiently diagnose these arrhythmias promptly. There exist many techniques for the detection of arrhythmias; however, the most widely adopted method is the use of an Electrocardiogram (ECG). The manual analysis of ECGs by medical experts is often inefficient. Therefore, the detection and recognition of ECG characteristics via machine-learning techniques have become prevalent. There are two major drawbacks of existing machine-learning approaches: (a) they require extensive training time; and (b) they require manual feature selection. To address these issues, this paper presents a novel deep-learning framework that integrates various networks by stacking similar layers in each network to produce a single robust model. The proposed framework has been tested on two publicly available datasets for the recognition of five micro-classes of arrhythmias. The overall classification sensitivity, specificity, positive predictive value, and accuracy of the proposed approach are 98.37%, 99.59%, 98.41%, and 99.35%, respectively. The results are compared with state-of-the-art approaches. The proposed approach outperformed the existing approaches in terms of sensitivity, specificity, positive predictive value, accuracy and computational cost.