Mitral valve segmentation is a crucial first step to establish a machine learning pipeline that can support practitioners into performing the diagnosis of mitral valve diseases, surgical planning, and intraoperative procedures. To this end, we propose a totally automated and unsupervised mitral valve segmentation algorithm, based on a neural network low-dimension matrix factorization of the echocardiography video. The method is evaluated in a collection of echocardiography video of patients with a variety of mitral valve diseases and exceeds the state-of-the-art method in all the metrics considered.