Introduction: Assessing internal validity of evidence on effectiveness from clinical trials is, in essence, a process of scientific inference for causality. As a cornerstone for evidence based decision making, many methods have been proposed for facilitating this process, and existing tools are getting ever more complicated. Method: This opinion article reviews the philosophy behind internal validity assessment, and critiqued a widely used tool (GRADE) for operationalizing this process. The following limitations of GRADE are discussed: (i) under-rating the importance of study design; (ii) inapplicability of GRADE to a single randomized controlled trial; and (iii) quality of evidence on direction and size of effect are not separated. Under this context a simpler scheme is proposed. Result: As an initial proposal for debate, we introduce a novel framework named the Chinese and Integrative Medicine Evidence RAting System (CHIMERAS). In CHIMERAS, direction of effect and effect size are rated separately. Four major aspects are considered in the appraisal of evidence quality: (i) study designs and the nature of outcomes; (ii) methodological quality of the systematic reviews reporting the outcomes; (iii) risk of publication bias; and (iv) effect size. Discussion: The feasibility of using CHIMERAS for appraising evidence from a systematic review on Chinese herbal medicine is piloted, and in the future this new system requires further evaluation to enhance its sophistication and implementability. This serves as a first step to open up debate on how evidence quality on Chinese and integrative medicine maybe assessed from a wider perspective.