The work presented in this paper is part of a system able to perform risk classification of patients based on medical image analysis and on the semantically structured information of patient data from medical records and biochemical data. More specifically, the paper focuses on Intravascular Ultrasound (IVUS) image processing and the automated segmentation developed to extract the useful arterial boundaries. This is coupled with the design and implementation of a semantic reasoning-enabled knowledge base in OWL that integrates data from heterogeneous sources and incorporates functionality for DL classification. Performance evaluation of both IVUS image processing and knowledge base is discussed.