Sea ice is one of the most prominent marine disasters in high latitudes. Remote sensing technology provides an effective means for sea ice detection. Remote sensing sea ice images contain rich spectral and spatial information. However, most traditional methods only focus on spectral information or spatial information, and do not excavate the feature of spectral and spatial simultaneously in remote sensing sea ice images classification. At the same time, the complex correlation characteristics among spectra and small sample problem in sea ice classification also limit the improvement of sea ice classification accuracy. For this issue, this paper proposes a new remote sensing sea ice image classification method based on squeeze-and-excitation (SE) network, convolutional neural network (CNN), and support vector machines (SVMs). The proposed method designs 3D-CNN deep network so as to fully exploit the spatial-spectrum features of remote sensing sea ice images and integrates SE-Block into 3D-CNN in-depth network in order to distinguish the contributions of different spectra to sea ice classification. According to the different contributions of spectral features, the weight of each spectral feature is optimized by fusing SE-Block in order to further enhance the sample quality. Finally, information-rich and representative samples are chosen by combining the idea of active learning and input into SVM classifier, and this achieves superior classification accuracy of remote sensing sea ice images with small samples. In order to verify the effectiveness of the proposed method, we conducted experiments on three different data from Baffin Bay, Bohai Bay, and Liaodong Bay. The experimental results show that compared with other classical classification methods, the proposed method comprehensively considers the correlation among spectral features and the small samples problems and deeply excavates the spatial-spectrum characteristics of sea ice and achieves better classification performance, which can be effectively applied to remote sensing sea ice image classification.