This study elaborates the “what works?” question by exploring the effects of variability in program implementation quality on achievement. Particularly, the effects on achievement of computerized assessment of reading were investigated, analyzing data on 51,000 students in Grades 1 – 12 who read over 3 million books. When minimum implementation quality criteria were met, the positive effect of computerized assessment was higher in the earlier grades and for lower achievement students. Implementation quality tended to decline at higher grade levels. With higher implementation quality, reading achievement gains were higher for students of all levels of achievement and across all grades, but especially in the higher grades. Very high gains and effect sizes were evident with very high implementation quality, particularly in Grades 1 – 4. Implications for practice, the interpretation of research, and policy are noted