Infrared and visible imaging play crucial roles in non-destructive testing, where accurate defect segmentation and detection are paramount. However, the scarcity of annotated training data or the limited number of data availability often poses a challenge. To address this, we propose an innovative framework tailored to the domain of infrared and visible imaging, integrating segmentation and detection tasks. The proposed approach eliminates the dependency on annotated defect data during training, enabling models to adapt to real-world scenarios with limited annotations. By utilizing super-pixel segmentation and texture analysis, the proposed method enhances the accuracy of defect detection. Concrete structures, globally subjected to aging and degradation, demand constant monitoring for structural health. Traditional manual crack detection methods are labor-intensive, necessitating automated systems. The proposed approach combines deep learning-based super-pixel segmentation with texture analysis, offering a solution for limited-defect-data situations. Utilizing convolutional neural networks (CNNs) for super-pixel segmentation and texture features for defect analysis, the proposed methodology improves the efficiency and accuracy of crack detection, especially in scenarios with limited labeled data or a limited number of data available. Evaluation on public benchmark datasets have validated the effectiveness of the proposed approach in detecting cracks in concrete structures.