The effectiveness of exercise to reduce pain and improve functioning in osteoarthritis of the knee (OAk) is well substantiated. Underlying mechanisms are still under debate and better understanding of the pathways involved may contribute to more targeted treatment strategies. The present qualitative analysis of the literature aims to provide an overview of theoretical models that are put forward to explain the beneficial treatment effects of exercise in OAk. An inductive qualitative approach, based on the 'grounded theory' of Glaser and Straus, was used. Twenty-two studies emphasizing on exercise therapy for OAk, collected from three Cochrane reviews and nine guidelines of the Physiotherapy Evidence Database (PEDRO) published between 2000 and 2012, were included. The introduction and discussion parts of these papers were screened for explanations of exercise-induced benefits in OAk patients. Seventy-three key points were identified which were subdivided into 16 core theoretical concepts. Finally, 5 categories were formed: neuromuscular, peri-articular, intra-articular, psychosocial components, and general fitness and health. We referred to scientific evidence that was used in the included studies to describe and categorize the concepts. Future research on exercise in OAk should allow distinguishing the contribution of different potential pathways to the treatment effects.