We segment atherosclerotic plaque components in in-vivo MRI and CT data using supervised voxelwise classification. The most reliable ground truth can be obtained from histology sections, however, it is not straightforward to use this for classifier training as the registration with in-vivo data often shows misalignments. Therefore, for training we incorporate uncertainty in the ground truth via "soft" labels that indicate a probability for each class. Soft labels are created by Gaussian blurring of the original hard segmentations, and weighted by the registration accuracy. Classification is evaluated on the relative volumes for fibrous, lipid-rich necrotic and calcified tissue. Using conventional "hard" labels, the differences between the ground truth and classification result per subject are 0.4±3.6% for calcification, 7.6±14.9% for fibrous and 7.2±14.5% for necrotic tissue. Using the new approach accuracy is improved: for calcification 0.6±1.6%, fibrous 3.6±16.8% and necrotic tissue 2.9±16.1%.