Background: Factors affecting morphological changes in the liver following selective internal radiation therapy (SIRT) are unclear, and the available literature focuses on non-anatomical volumetric assessment techniques in a lobar treatment setting. This study aimed to investigate quantitative changes in the liver post-SIRT using an anatomical volumetric approach in hepatocellular carcinoma (HCC) patients with different levels of treatment selectivity and evaluate the parameters affecting those changes. This retrospective, single-institution, IRB-approved study included 88 HCC patients. Whole liver, liver segments, tumor burden, and spleen volumes were quantified on MRI at baseline and 3/6/12 months post-SIRT using a segmentation-based 3D software relying on liver vascular anatomy. Treatment characteristics, longitudinal clinical/laboratory, and imaging data were analyzed. The Student’s t-test and Wilcoxon test evaluated volumetric parameters evolution. Spearman correlation was used to assess the association between variables. Uni/multivariate analyses investigated factors influencing untreated liver volume (uLV) increase. Results: Most patients were cirrhotic (92%) men (86%) with Child–Pugh A (84%). Absolute and relative uLV kept increasing at 3/6/12 months post-SIRT vs. baseline (all, p ≤ 0.005) and was maximal during the first 6 months. Absolute uLV increase was greater in Child–Pugh A5/A6 vs. ≥B7 at 3 months (A5, p = 0.004; A6, p = 0.007) and 6 months (A5, p = 0.072; A6, p = 0.031) vs. baseline. When the Child–Pugh class worsened at 3 or 6 months post-SIRT, uLV did not change significantly, whereas it increased at 3/6/12 months vs. baseline (all p ≤ 0.015) when liver function remained stable. The Child–Pugh score was inversely correlated with absolute and relative uLV increase at 3 months (rho = −0.21, p = 0.047; rho = −0.229, p = 0.048). In multivariate analysis, uLV increase was influenced at 3 months by younger age (p = 0.013), administered 90Y activity (p = 0.003), and baseline spleen volume (p = 0.023). At 6 months, uLV increase was impacted by younger age (p = 0.006), whereas treatment with glass microspheres (vs. resin) demonstrated a clear trend towards better hypertrophy (f = 3.833, p = 0.058). The amount (percentage) of treated liver strongly impacted the relative uLV increase at 3/6/12 months (all f ≥ 8.407, p ≤ 0.01). Conclusion: Liver function (preserved baseline and stable post-SIRT) favored uLV hypertrophy. Younger patients, smaller baseline spleen volume, higher administered 90Y activity, and a larger amount of treated liver were associated with a higher degree of untreated liver hypertrophy. These factors should be considered in surgical candidates undergoing neoadjuvant SIRT.