This work addressed a major clinical challenge, namely valid treatment planning for carotid atherosclerosis (CA). To this end, it introduced a novel computer-aided-diagnosis (CAD) scheme, which relies on the analysis of ultrasound videos to stratify patient risk. Based on Hidden Markov Models (HMM), it is guided by spatiotemporal patterns representing motion and strain activity in the arterial wall and it acts as a voice-recognition analogue. The designed CAD scheme was optimized and evaluated on a dataset of 96 high-and low-risk patients with CA, by investigating patterns with the strongest discrimination power and the optimal HMM parameterization. It was concluded that the optimized CAD scheme provides a CAD response with accuracy between 76% and 79%. The introduced CAD scheme may serve as a valuable tool in the routine clinical practice for CA toward personalized and valid therapeutic decision for the disease.