In this paper, we develop a computer aided diagnosis algorithm to detect and classify the abnormalities in vision-based endoscopic examination. We focus on analyzing the traditional gastroscope data and help the medical experts improve the accuracy of medical diagnosis with our analysis tool. To achieve this, we first segment the image into superpixels, then extract various color and texture features from them and combine the features into one feature vector to represent the images. This approach is more flexible and accurate than the traditional patch-based image representation. Then we design a novel feature selection model with group sparsity, Deep Sparse SVM (DSSVM) that not only can assign a suitable weight to the feature dimensions like the other traditional feature selection models, but also directly exclude useless features from the feature pool. Thus, our DSSVM model can maintain the accuracy while reducing the computation complexity. Moreover, the image quality is also pre-assessed. For the experiments, we build a new gastroscope dataset with a total of about 3800 images from 1284 volunteers, and conducted various experiments and comparisons with other algorithms to justify the effectiveness and efficiency of our algorithm.