Controlled exposure experiments that measure animal response to vessels can inform relevant wildlife-viewing guidelines and reveal how they make decisions about changes in their environment. Previous experimental studies documented stereotyped avoidance responses by killer whales to boats. Additional observations collected during these studies showed an apparent shift in avoidance behaviour at high traffic levels. Our study tested experimentally whether whales did respond differently to approach by few (1–3) versus many (>3) vessels. Data were collected in summer 2004 in Johnstone Strait, British Columbia, using a theodolite to track the positions of boats and individually identifiable focal whales during control and treatment (few vs. many boats) phases. The responses of 16 adult male killer whales differed significantly between treatment levels (Wilcoxon's test, P=0.0148). Swimming paths became more tortuous when few boats approached whales, but straighter as many boats approached. Pooling treatments would have masked significant responses with high statistical confidence (Wilcoxon's test, P>0.999), falsely suggesting that boat presence had no effect. The division between few and many boats was supported by 140 opportunistic observations on 26 whales from a population of 216. We used generalized additive models to control for the effects of confounding variables, detected a non-linear relationship between number of boats and whales' swimming path directness and confirmed an inflection point at approximately three boats within 1000 m. We urge caution when designing controlled exposure assessments that rely on a simple absence–presence framework, which can mask multivariate or non-linear responses. Experimental design, coupled with analytical techniques incorporating statistical power and appropriateness of treatments and response variables, must be considered when interpreting the biological significance of null findings from impact assessments. Our study provides new information about levels of habitat degradation that this marine apex predator can tolerate.