Recent advances in computer vision have allowed broad applications in every area of life, and agriculture is not left out. For the agri-food industry, the use of advanced technology is essential. Owing to deep learning’s capability to learn robust features from images, it has witnessed enormous application in several fields. Fruit detection and classification remains challenging due to the form, color, and texture of different fruit species. While studying the impact of computer vision on fruit detection and classification, we pointed out that till 2018 many conventional machine learning methods were utilized while a few methods exploited the application of deep learning methods for fruit detection and classification. This has prompted us to pursue an extensive study on surveying and implementing deep learning models for fruit detection and classification. In this article, we intensively discussed the datasets used by many scholars, the practical descriptors, the model’s implementation, and the challenges of using deep learning to detect and categorize fruits. Lastly, we summarized the results of different deep learning methods applied in previous studies for the purpose of fruit detection and classification. This review covers the study of recently published articles that utilized deep learning models for fruit identification and classification. Additionally, we also implemented from scratch a deep learning model for fruit classification using the popular dataset “Fruit 360” to make it easier for beginner researchers in the field of agriculture to understand the role of deep learning in the agriculture domain.