Brief published guides
- A guide to critical appraisal of evidenceFineout-Overholt, E. (2019).
Nursing Critical Care 14(3), 24-30.
https://doi.org/10.1097/01.CCN.0000554830.12833.2f - Determining the level of evidence: Experimental research appraisalGlasofer, A. & Townsend, A.B. (2019).
Nursing Critical Care 14(6), 22-25.
https://doi.org/10.1097/01.CCN.0000580120.03118.1d - Determining the level of evidence: Nonexperimental research designsGlasofer, A & Townsend, A.B. (2020).
Nursing Critical Care 15(1): 24-27.
https://doi.org/10.1097/01.CCN.0000612856.94212.9b - Determining the level of evidence: Nonresearch evidenceGlasofer, A. & Townsend, A.B. (2020).
Nursing Critical Care 15(2), 22-26.
https://doi.org/10.1097/01.CCN.0000654792.71629.00
Critical Appraisal Tools
Tools for several different study types are available (including systematic reviews):
- List of appraisal tools by type of studyFrom Cardiff University. Includes the links above as well as others, but they categorize all the tools by the type(s) of study.
Tools that emphasize a particular type of study:
- AGREE ToolsFor development, reporting and evaluation of practice guidelines and health system guidance. Read the descriptions to discover the most appropriate tool(s) for your need.
- AMSTAR 2Appraise systematic reviews
- Cochrane Risk of BiasRoB 2 - Risk of bias in randomized trials
ROBINS-I - Risk of bias in non-randomized studies
ROB-ME - Risk of bias due to missing evidence in a synthesis (preliminary version of the tool) - CONSORTAppraise parallel group randomized trials
Main Evaluation/Synthesis Table
This table should list all the sources that comprise your body of evidence and indicate Author (Year), Conceptual Framework, Design and/or Method Sample and/or Setting, Major Variables Studied (and Their Definitions), Measurement, Data Analysis, Findings, and Appraisal (Worth to Practice).
Be concise. Abbreviate heavily and be sure to include a legend at the bottom for abbreviations.
- Evidence Evaluation Table TemplateCopy and paste from this template to create the columns for your own Evidence Evaluation Table. [Online-only content for “Critical Appraisal of the Evidence: Part I,” by Fineout-Overholt and colleagues in the American Journal of Nursing, July 2010, p. 47-52]
- Sample Evidence Evaluation Table - CompletedThis is an example of an Evidence Evaluation with all of the elements filled in. [Online-only content for “Critical Appraisal of the Evidence: Part III,” by Fineout-Overholt and colleagues in the American Journal of Nursing,
November 2010, p. 43-51.]
Additional Synthesis Tables (for easy visual summaries)
- Full article: Critical appraisal of the evidence, Part III"It’s not the number of studies or projects that determines
the reliability of their findings, but the uniformity and
quality of their methods" (p. 46).
See Table 2 for an example of the Body of Evidence Table and Table 3 for an example of a Data Extraction Table.. Note: For Data Extraction Table, Arrows may be used for increases and decreases. Include footnotes to add important information about each study (such as level of evidence and statistical significance). Table may also include color-coding for quick interpretation (for example, green for good; red for bad), but color should not be the only way meaning is conveyed.
Other Methods of Assessing the Strength of the Evidence
These methods are often used with systematic reviews.
GRADE
GRADE CERQual
- GRADE CERQual(Confidence in the Evidence from Reviews of Qualitative Research): GRADE for qualitative systematic reviews to assess the evidence or develop recommendations.
ConQual
- ConQual ApproachRating the confidence of qualitative systematic review findings. An article describing the approach.
EPC (AHRQ)
- Grading the strength of a body of evidence when assessing health care interventions: an EPC update. (2015)."Updated guidance continues to be conceptually similar to GRADE."
Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)
- PRISMA Key Documentschecklist, flow diagram, statement, and explanation
- PRISMA aims to help authors improve the reporting of systematic reviews and meta-analyses.
- PRISMA may also be useful for critical appraisal of published systematic reviews, although it is not a quality assessment instrument to gauge the quality of a systematic review.
Check out these short videos from Towson University's Health Professions Librarian Carrie Price for more information about PRISMA:
Journal Credibility Questions in Evidence Synthesis
Journals that follow deceptive or fraudulent publication practices are often found alongside more legitimate publications, whether in Google Scholar or established library databases.
This study indicated that evidence synthesizers tend to advocate for having a process in place to identify predatory journals. They supported the need to assess the quality of the data published in them and indicated that only good quality studies should be incorporated into the synthesis.
When synthesizing evidence, you should have a clear protocol for how you will identify potentially predatory journals, how you will evaluate quality, and to what extent you will include studies from those publications.
- How should we handle predatory journals in evidence synthesis? A descriptive survey-based cross-sectional study of evidence synthesis expertsBarker, T. H., Pollock, D., Stone, J. C., Klugar, M., Scott, A. M., Stern, C., Wiechula, R., Shamseer, L., Aromataris, E., Ross, W. A., & Munn, Z. (2023). How should we handle predatory journals in evidence synthesis? A descriptive survey‐based cross‐sectional study of evidence synthesis experts. Research Synthesis Methods, 14(3), 370–381. https://doi.org/10.1002/jrsm.1613
Avoiding Retracted Articles in Evidence Synthesis
Retraction aims to correct both fraud and unintended errors in the scientific record. Including retracted literature in evidence syntheses can cause patient harm. However, retraction notices are not always clearly publicized, so retracted articles often still get cited post-retraction.
This opinion piece calls for more careful scrutiny by authors, editors, and peer reviewers to avoid citing retracted articles.
- How to Stop the Unknowing Citation of Retracted Papers (editorial)Marcus, A., Abritis, A. J., & Oransky, I. (2022). How to Stop the Unknowing Citation of Retracted Papers. Anesthesiology, 137(3), 280–282.
