RationaleLabel‐free quantification (LFQ) is a popular strategy for shotgun proteomics. A variety of LFQ algorithms have been developed recently. However, a comprehensive comparison of the most commonly used LFQ methods is still rare, in part due to a lack of clear metrics for their evaluation and an annotated and quantitatively well‐characterized data set.MethodsFive LFQ methods were compared: spectral counting based algorithms SI_N, emPAI, and NSAF, and approaches relying on the extracted ion chromatogram (XIC) intensities, MaxLFQ and Quanti. We used three criteria for performance evaluation: coefficient of variation (CV) of protein abundances between replicates; analysis of variance (ANOVA); and the root‐mean‐square error of logarithmized calculated concentration ratios, referred to as standard quantification error (SQE). Comparison was performed using a quantitatively annotated publicly available data set.ResultsThe best results in terms of inter‐replicate reproducibility were observed for MaxLFQ and NSAF, although they exhibited larger standard quantification errors. Using NSAF, all quantitatively annotated proteins were correctly identified in the Bonferronni‐corrected results of the ANOVA test. SI_N was found to be the most accurate in terms of SQE. Finally, the current implementations of XIC‐based LFQ methods did not outperform the methods based on spectral counting for the data set used in this study.ConclusionsSurprisingly, the performances of XIC‐based approaches measured using three independent metrics were found to be comparable with more straightforward and simple MS/MS‐based spectral counting approaches. The study revealed no clear leader among the latter. Copyright © 2017 John Wiley & Sons, Ltd.