AbstractDeep learning is a promising method for medical image analysis because it can automatically acquire meaningful representations from raw data. However, a technical challenge lies in the difficulty of determining which types of internal representation are associated with a specific task, because feature vectors can vary dynamically according to individual inputs. Here, based on the magnetic resonance imaging (MRI) of gliomas, we propose a novel method to extract a shareable set of feature vectors that encode various parts in tumor imaging phenotypes. By applying vector quantization to latent representations, features extracted by an encoder are replaced with a fixed set of feature vectors. Hence, the set of feature vectors can be used in downstream tasks as imaging markers, which we call deep radiomics. Using deep radiomics, a classifier is established using logistic regression to predict the glioma grade with 90% accuracy. We also devise an algorithm to visualize the image region encoded by each feature vector, and demonstrate that the classification model preferentially relies on feature vectors associated with the presence or absence of contrast enhancement in tumor regions. Our proposal provides a data-driven approach to enhance the understanding of the imaging appearance of gliomas.