Deep neural networks are efficient methods of recognizing image patterns and have been largely implemented in computer vision applications. Object detection has many applications in computer vision, including face and vehicle detection, video surveillance, and plant leaf detection. An automatic flower identification system over categories is still challenging due to similarities among classes and intraclass variation, so the deep learning model requires more precisely labeled and high-quality data. In this proposed work, an optimized and generalized deep convolutional neural network using Faster-Recurrent Convolutional Neural Network (Faster-RCNN) and Single Short Detector (SSD) is used for detecting, localizing, and classifying flower objects. We prepared 2000 images for various pretrained models, including ResNet 50, ResNet 101, and Inception V2, as well as Mobile Net V2. In this study, 70% of the images were used for training, 25% for validation, and 5% for testing. The experiment demonstrates that the proposed Faster-RCNN model using the transfer learning approach gives an optimum mAP score of 83.3% with 300 and 91.3% with 100 proposals on ten flower classes. In addition, the proposed model could identify, locate, and classify flowers and provide essential details that include flower name, class classification, and multilabeling techniques.