In this review, comparisons between organic and conventional cropping systems are discussed. Publications from four topics, crop yields, carbon sequestration, biological diversity and nitrogen leaching were selected as examples to point out pitfalls and shortcomings in comparative analysis that can weaken or even disqualify evaluations. Inconsistent results between different comparative studies were found to be pseudo-contradictions. As the experimental design of comparative organic and conventional cropping systems often is biased in some aspects, suitable denominators for comparative assessment are discussed (ratios per area, per product and per land demand for the same amount of product). Conditions for equitable evaluations are outlined in order to avoid biased design, inappropriate interpretations and flawed conclusions. We stress that respecting at least three stringency criteria will help to ensure the scientific quality of data interpretation of comparative studies: similar soil fertility status at start, comparable type of crop production, and quantification of off-farm organic and nutrient input.