Background: Accurate clinical assessment in multiple sclerosis (MS) is challenging. The Assess MS system is being developed to automatically quantify motor dysfunction in MS, including upper extremity function (UEF) and mobility. Objective: To determine to what extent combinations of standardized movements included in the Assess MS system explain accepted measures of UEF and mobility. Methods: MS patients were recruited at four European MS centres. Eight movements were selected, including tasks of activities of daily living (ADL) and classical neurological tests. Movements were recorded on video and rated by experienced neurologists ( n = 5). Subsequently, multivariate linear regression models were performed to explain the variance of the Nine-Hole Peg Test (9HPT), Arm Function in Multiple Sclerosis Questionnaire (AMSQ) and Timed-25 Foot Walk test (T25WT). Results: In total, 257 patients were included. The movements explained 62.9% to 80.1% of the variance of the 9HPT models, 43.3% and 44.3% of the AMSQ models and 70.8% of the T25WT. In all models, tasks of ADL contributed most to the variance. Conclusion: Combinations of movements are valuable to assess UEF and mobility. Incorporating ADL tasks into daily clinical practice and clinical trials may be more valuable than the classical neurological examination of UEF and mobility.