The paddy crop is the most essential and consumable agricultural produce. Leaf disease impacts the quality and productivity of paddy crops. Therefore, tackling this issue as early as possible is mandatory to reduce its impact. Consequently, in recent years, deep learning methods have been essential in identifying and classifying leaf disease. Deep learning is used to observe patterns in disease in crop leaves. For instance, organizing a crop’s leaf according to its shape, size, and color is significant. To facilitate farmers, this study proposed a Convolutional Neural Networks-based Deep Learning (CNN-based DL) architecture, including transfer learning (TL) for agricultural research. In this study, different TL architectures, viz. InceptionV3, VGG16, ResNet, SqueezeNet, and VGG19, were considered to carry out disease detection in paddy plants. The approach started with preprocessing the leaf image; afterward, semantic segmentation was used to extract a region of interest. Consequently, TL architectures were tuned with segmented images. Finally, the extra, fully connected layers of the Deep Neural Network (DNN) are used to classify and identify leaf disease. The proposed model was concerned with the biotic diseases of paddy leaves due to fungi and bacteria. The proposed model showed an accuracy rate of 96.4%, better than state-of-the-art models with different variants of TL architectures. After analysis of the outcomes, the study concluded that the anticipated model outperforms other existing models.