AbstractThe aim of this study is to evaluate whether we could develop a machine learning method to distinguish models of cerebrospinal fluid shunt valves (CSF-SVs) from their appearance in clinical X-rays. This is an essential component of an automatic MRI safety system based on X-ray imaging. To this end, a retrospective observational study using 416 skull X-rays from unique subjects retrieved from a clinical PACS system was performed. Each image included a CSF-SV representing the most common brands of programmable shunt valves currently used in US which were split into five different classes. We compared four machine learning pipelines: two based on engineered image features (Local Binary Patterns and Histogram of Oriented Gradients) and two based on features learned by a deep convolutional neural network architecture. Performance is evaluated using accuracy, precision, recall and f1-score. Confidence intervals are computed with non-parametric bootstrap procedures. Our best performing method identified the valve type correctly 96% [CI 94–98%] of the time (CI: confidence intervals, precision 0.96, recall 0.96, f1-score 0.96), tested using a stratified cross-validation approach to avoid chances of overfitting. The machine learning pipelines based on deep convolutional neural networks showed significantly better performance than the ones based on engineered image features (mean accuracy 95–96% vs. 56–64%). This study shows the feasibility of automatically distinguishing CSF-SVs using clinical X-rays and deep convolutional neural networks. This finding is the first step towards an automatic MRI safety system for implantable devices which could decrease the number of patients that experience denials or delays of their MRI examinations.