Abstract The use of sexed semen at dairy farms has improved heifer replacement over the last decade by allowing greater control over the number of retained females and enabling the selection of dams with superior genetics. Alternatively, beef semen can be used in genetically inferior dairy cows to produce crossbred (beef x dairy) animals that can be sold at a higher price. Although crossbreeding became profitable for dairy farmers, meat cuts from beef x dairy crosses often lack quality and shape uniformity. Technologies for quickly predicting carcass traits for animal grouping before harvest may improve meat cut uniformity in crossbred cattle. Our objective was to develop a deep learning approach for predicting ribeye area and circularity of live animals through 3D body surface images using two neural networks: 1) nested Pyramid Scene Parsing Network (nPSPNet) for extracting features and 2) Convolutional Neural Network (CNN) for estimating ribeye area and circularity from these features. A group of 56 calves were imaged using an Intel RealSense D435 camera. A total of 327 depth images were captured from 30 calves and labeled with masks outlining the calf body to train the nPSPNet for feature extraction. Additional 42,536 depth images were taken from the remaining 26 calves along with three ultrasound images collected for each calf from the 12/13th ribs. The ultrasound images (three by calf) were manually segmented to calculate the average ribeye area and circularity and then paired with the depth images for CNN training. We implemented a nested cross-validation approach, in which all images for one calf were removed (leave-one-out, LOO), and the remaining calves were further divided into training (70%) and validation (30%) sets within each LOO iteration. The proposed model predicted ribeye area with an average coefficient of determination (R2) of 0.74% and 7.3% mean absolute error of prediction (MAEP) and the ribeye circularity with an average R2 of 0.87% and 2.4% MAEP. Our results indicate that computer vision systems could be used to predict ribeye area and circularity in live animals, allowing optimal management decisions toward smart animal grouping in beef x dairy crosses and purebred.