Background Specialist care units cater to targeted cohorts of patients, applying evidence-based practice to people with a specific condition (e.g., dementia) or meeting other specific criteria (e.g., children). This paper aimed to collate perceptions of local consumers and health providers around specialist care units, as a model of care that may be considered for a new local healthcare facility. Methods This was a qualitative study using two-hour workshops and interviews to collect data. Participants were consumers and health providers in the planned facility’s catchment: 49 suburbs in metropolitan Australia. Consumers and health providers were recruited through advertisements and emails. An initial survey collected demographic details. Consumers and health providers participated in separate two-hour workshops in which a scenario around the specialist unit model was presented and discussion on benefits, barriers and enablers of the model was led by researchers. Detailed notes were taken for analysis. Results Five consumer workshops (n = 22 participants) and five health provider workshops (n = 42) were conducted. Participants were representative of this culturally diverse region. Factors identified by participants as relevant to the specialist unit model of care included: accessibility; a perceived narrow scope of practice; coordination with other services; resources and infrastructure; and awareness and expectations of the units. Some factors identified as risks or barriers when absent were identified as strengths and enablers when present by both groups of participants. Conclusions Positive views of the model centred on the higher perceived quality of care received in the units. Negative views centred on a perceived narrow scope of care and lack of flexibility. Consumers hinted, and providers stated explicitly, that the model needed to be complemented by an integrated model of care model to enable continuity of care and easy transfer of patients into and out of the specialist unit.