This study addresses the poorly explored issue of the control of false positive rate (FPR) in the mapping of pair-wise epistatic quantitative trait loci (QTL). A nested test framework was developed to (1) allow pre-identified QTL to be used directly to detect epistasis in one-dimensional genome scans, (2) to detect novel epistatic QTL pairs in two-dimensional genome scans and (3) to derive genome-wide thresholds through permutation and handle multiple testing. We used large-scale simulations to evaluate the performance of both the one- and two-dimensional approaches in mapping different forms and levels of epistasis and to generate profiles of FPR, power and accuracy to inform epistasis mapping studies. We showed that the nested test framework and genome-wide thresholds were essential to control FPR at the 5% level. The one-dimensional approach was generally more powerful than the two-dimensional approach in detecting QTL-associated epistasis and identified nearly all epistatic pairs detected from the two-dimensional approach. However, only the two-dimensional approach could detect epistatic QTL with weak main effects. Combining the two approaches allowed effective mapping of different forms of epistasis, whereas using the nested test framework kept the FPR under control. This approach provides a good search engine for high-throughput epistasis analyses.