Abstract Background Age-related deposition of fat in skeletal muscle is associated with functional limitations. Skeletal muscle fat may be present in people with preserved muscle mass or accompanied by muscle wasting. However, it is not clear if the association between muscle fat deposition and physical performance is moderated by muscle mass. Objective To determine whether the association between midthigh intermuscular fat and physical performance is moderated by muscle area. Methods We performed a cross-sectional analysis of the Health, Aging, and, Body Composition (ABC) study data collected in 2002–2003 (n = 1897, women: 52.2%). Midthigh muscle cross-sectional area (by computed tomography) and physical performance measures were compared across quartiles of intermuscular fat absolute area. Moderation analysis was performed to determine the conditional effect of intermuscular fat on physical performance as a function of muscle area. Conditional effects were evaluated at three levels of muscle area (mean and ± 1 standard deviation [SD]; 213.2 ± 53.2 cm2). Results Simple slope analysis showed that the negative association between intermuscular fat area (cm2) and leg strength (N·m) was of greater magnitude (beta coefficient [b], 95% confidence interval [CI] = −0.288 [−0.427, −0.148]) in participants with greater muscle area (ie, 1 SD above the mean) compared to those with lower muscle area (ie, at mean [b = −0.12 {−0.248, 0.008}] or 1 SD below the mean [b = 0.048 {−0.122, 0.217}]). Similarly, the negative association of intermuscular fat with 400-m walk speed (m/s) and chair stand (seconds) was greater in those with higher muscle areas (p < .001) compared to those with lower muscle areas. Conclusions The association between higher intermuscular fat area and impaired physical function in aging is moderated by muscle area.