Fitness centres are an obvious arena for performing physical activity for the general population but representation of adults with physical disabilities (AwPD) is lacking. To increase possibilities for AwPD to exercise in fitness centres together with adults without physical disabilities (AwoPD), the aim of this study was to identify, synthesise, and compare barriers to, and facilitators of, exercising in fitness centres for each group. A scoping review was conducted and data extraction of the barriers and facilitators was performed independently by two researchers on six categories of contextual factors based on the framework of Di Blasi: (1) The fitness centre setting; (2) The fitness centre user characteristics; (3) The fitness instructor/staff characteristics; (4) The fitness centre user–instructor/management relationship; and (5) The fitness/exercise characteristics. An extra category, (6) Other relationships, was added. The PRISMA Extension for Scoping Reviews was used for reporting. Of the 102 included papers, only 26 (25%) of the papers were on AwPD, which focused mainly on physical barriers (category 1: inaccessible settings). In contrast, the remaining 76 papers involving AwoPD focused primarily on facilitators (category 2: motivational factors and exercising effects). In categories 3–6, the two groups had similar results, as both groups preferred skilled instructors, a welcoming and comfortable fitness centre environment, an ability to exercise at their preferred type and level, and good social connections. Since most data were based on AwoPD, more studies on actual experiences from AwPD are needed, to reveal the facilitators/motivational factors for fitness centre use.