Abstract Adopting Weir’s (2005) socio-cognitive validation framework, the present paper focuses on the syntactic and lexical complexity of listening comprehension subtests in three B2-level examinations: The City Guilds international examination in English, The First Certificate in English, and the General Matura in English. By analysing and interpreting the results obtained from different automated tools, the research aims to determine to what extent the three subtests are comparable. The results of the study suggest the unreliability of the Common European Framework of Reference for Languages (CEFR) as a sole mechanism for test comparisons.