Horticulture crops take a crucial part of the Indian economy by creating employment, supplying raw materials to different food processing industries. Mangoes are one of the major crops in horticulture. General Infections in Mango trees are common by various climatic and fungal infections, which became a cause for reducing the quality and quantity of the mangos. The most common diseases with bacterial infection are anthracnose and Powdery Mildew. In recent years, it has been perceived that different variants of deep learning architectures are proposed for detecting and classifying the problems in the agricultural domain. The Convolutional Neural Network (CNN) based architectures have performed amazingly well for disease detection in plants but at the same time lacks rotational or spatial invariance. A relatively new neural organization called Capsule Network (CapsNet) addresses these limitations of CNN architectures. Hence, in this work, a variant of CapsNet called Multilevel CapsNet is introduced to characterize the mango leaves tainted by the anthracnose and powdery mildew diseases. The proposed architecture of this work is validated on a dataset of mango leaves collected in the natural environment. The dataset comprises both healthy and contaminated leaf pictures. The test results approved the undeniable level of exactness of the proposed framework for the characterization of mango leaf diseases with an accuracy of 98.5%. The outcomes conceive the higher-order precision of the proposed Multi-level CapsNet model when contrasted with the other classification algorithms such as Support Vector Machine (SVM) and CNNs.