Abstract Background Policy makers in Africa are ambivalent about the need for family physicians to strengthen district health services. Evidence on the impact of family physicians is therefore needed. The aim was to develop a tool to evaluate the impact of family physicians on district health services according to the six expected roles that have been defined nationally. Methods Mixed methods were used to develop, validate, pilot and test the reliability of the tool in the Western Cape Province, South Africa. An expert panel validated the content and construction of the tool. The tool was piloted by 94 respondents who evaluated eight family physicians. Cronbach alpha scores were calculated to test the reliability of the tool. The impact of these family physicians in the pilot study was also analysed. Results A draft tool was successfully developed, validated, and proved reliable (Cronbach alpha >0.8). The overall scores (scale of 1–4) were: Care provider = 3.5, Consultant = 3.4, Leader and champion of clinical governance = 3.4, Capacity builder = 3.3, Clinical trainer and supervisor = 3.2 and Champion of community-orientated primary care (COPC) = 3.1. The impact on COPC was significantly less than the impact of other roles (p