Highlights
- •One-third of systematic reviews in rehabilitation assess the certainty of evidence
- •The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) was the most common approach used
- •High uptake of approaches such as GRADE is recommended
Abstract
Objective
To determine how many systematic reviews (SRs) of the literature in rehabilitation
assess the certainty of evidence (CoE) and how many apply the Grading of Recommendations
Assessment, Development and Evaluation (GRADE) system to do this.
Data Sources
For this meta-research study, we searched PubMed and Cochrane Database of Systematic
Reviews databases for SRs on rehabilitation published in 2020.
Study Selection and Data Extraction
Two reviewers independently selected the SRs and extracted the data. Reporting characteristics
and appropriate use of the GRADE system were assessed.
Data Synthesis
The search retrieved 827 records: 29% (239/827) SRs evaluated CoE, 68% (163/239) of
which applied the GRADE system. GRADE was used by SRs of randomized controlled trials
(RCTs, 88%; 144/163), non-randomized intervention studies (NRIS, 2%; 3/163), and both
RCT and NRIS (10%; 16/163). In the latter case, a separate GRADE assessment according
to the study design was not provided in 75% (12/16). The reasons for GRADE judgment
were reported in 82% (134/163) of SRs.
Conclusions
One-third of SRs in rehabilitation assessed CoE with the GRADE system. GRADE assessment
was presented transparently by most SRs. Journal editors and funders should encourage
the uptake of the GRADE system when considering SRs in rehabilitation for publication.
The authors should pre-define GRADE assessment in a registered and/or published protocol.
Keywords
List of abbreviations:
CoE (certainty of evidence), GRADE (Grading of Recommendations Assessment, Development, and Evaluation), JIF (journal impact factor), NRIS (non-randomized intervention studies), OSF (Open Science Framework), PERSiST (implementing Prisma in Exercise, Rehabilitation, Sport medicine and SporTs science), PRISMA (Preferred Reporting Items for SRs and Meta-analyses), RCT (randomized controlled trials), ROBINS-I (Risk Of Bias In Non-Randomized Studies of Interventions), SIGN (Scottish Intercollegiate Guidelines Network), SoF (summary of findings)To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Archives of Physical Medicine and RehabilitationAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses.Lancet. 1999; 354: 1896-1900
- The GRADE approach, Part 1: how to assess the certainty of the evidence.Medwave. 2021; 21: e8109
- GRADE guidelines: 3. Rating the quality of evidence.J Clin Epidemiol. 2011; 64: 401-406
- Rating the quality of a body of evidence on the effectiveness of health and social interventions: a SR and mapping of evidence domains.Res Synth Methods. 2018; 9: 224-242
- What is "quality of evidence" and why is it important to clinicians?.BMJ. 2008; 336: 995-998
- GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables.J Clin Epidemiol. 2011; 64: 383-394
- GRADE Working Group. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations.BMJ. 2008; 336: 924-926
- GRADE guidelines: 4. Rating the quality of evidence–study limitations (risk of bias).J Clin Epidemiol. 2011; 64: 407-415
- GRADE guidelines: 9. Rating up the quality of evidence.J Clin Epidemiol. 2011; 64: 1311-1316
- [GRADE guidelines: 1. Introduction - GRADE evidence profiles and summary of findings tables].Z Evid Fortbild Qual Gesundhwes. 2012; 106: 357-368
- GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence.J Clin Epidemiol. 2019; 111: 105-114
Schünemann H, Brożek J, Guyatt G, Oxman A, editors. GRADE handbook for grading quality of evidence and strength of recommendations. Updated October 2013. The GRADE Working Group, 2013. Available at: guidelinedevelopment.org/handbook.
- PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting SRs.BMJ. 2021; 372: n160
- The PRISMA 2020 statement: an updated guideline for reporting SRs.BMJ. 2021; 372: n71
- What is a meta-epidemiological study? Analysis of published literature indicated heterogeneous study designs and definitions.J Comp Eff Res. 2020; 9: 497-508
- Methodological research: open questions, the need for 'research on research' and its implications for evidence-based health care and reducing research waste.Int J Evid Based Healthc. 2019; 17: 145-146
- Reporting of methodological studies in health research: a protocol for the development of the MethodologIcal STudy reportIng Checklist (MISTIC).BMJ Open. 2020; 10e040478
- Guidelines for reporting meta-epidemiological methodology research.Evid Based Med. 2017; 22: 139-142
- Clarifying differences between review designs and methods.Syst Rev. 2012; 1: 28
- PRESS peer review of electronic search strategies: 2015 guideline statement.J Clin Epidemiol. 2016; 75: 40-46
- 2015 Guideline Explanation and Elaboration (PRESS E&E).CADTH, Ottawa2016
- EndNote X9 version.Clarivate, Philadelphia, PA2013
- Rayyan-a web and mobile app for SRs.Syst Rev. 2016; 5: 210
- AMSTAR 2: a critical appraisal tool for SRs that include randomised or non-randomised studies of healthcare interventions, or both.BMJ. 2017; 358: j4008
- Meta-analyses frequently pooled different study types together: a meta-epidemiological study.J Clin Epidemiol. 2020; 118: 18-28
- Structural approach to bias in meta-analyses.Res Synth Methods. 2011; 2: 223-237
- Beyond causality: additional benefits of randomized controlled trials for improving health care delivery.Milbank Q. 2021; 99: 864-881
- The Cochrane Collaboration's tool for assessing risk of bias in randomised trials.BMJ. 2011; 343: d5928
- Issues relating to confounding and meta-analysis when including non-randomized studies in SRs on the effects of interventions.Res Synth Methods. 2013; 4: 26-35
- Stratification by quality induced selection bias in a meta-analysis of clinical trials.J Clin Epidemiol. 2019; 107: 51-59
- Stata Statistical Software: Release 15.Stata- Corp LLC, College Station, TX, USA2017
E-bRO. Evidence-Based Guideline Development Manual for Working Group members Original Publication. 2005. Available at: http://www.ha-ring.nl/download/literatuur/EBRO_handl_totaal.pdf. Accessed February 20, 2022.
Howick J, Chalmers I, Glasziou P, et al. “Explanation of the 2011 Oxford Centre for Evidence-Based Medicine (OCEBM) Levels of Evidence (Background Document)”. Oxford Centre for Evidence-Based Medicine. Available at: https://www.cebm.ox.ac.uk/resources/levels-of-evidence/ocebm-levels-of-evidence. Accessed February 20, 2022.
- Joanna Briggs Institute reviewers’ manual.The Joanna Briggs Institute, Adelaide2014
- Evidence-based practice and occupational therapy.Br J Occup Ther. 1997; 60: 474-478
- Extending an evidence hierarchy to include topics other than treatment: revising the Australian 'levels of evidence'.BMC Med Res Methodol. 2009; 9: 34
- The Scottish Intercollegiate Guidelines Network (SIGN): an update.Scott Med J. 2005; 50: 51-52
U.S. Preventive Services Task Force. U.S. Preventive Services Task Force ratings. Available at: https://www.uspreventiveservicestaskforce.org/Page/Name/us-preventive-services-task-force-ratings. Accessed February 20, 2022.
- GRADE: an emerging consensus on rating quality of evidence and strength of recommendations.BMJ. 2008; 336: 924-926
Joanna Briggs Institute. Supporting document for the Joanna Briggs Institute levels of evidence and grades of recommendations. 2014. The Joanna Briggs Institute Levels of Evidence and Grades of Recommendations Working Party. Available at: https://jbi.global/sites/default/files/2019-05/JBI-Levels-of-evidence_2014_0.pdf. Accessed October 26, 2016.
- Grading the strength of a body of evidence when assessing health care interventions for the effective health care program of the agency for healthcare research and quality: an update.AHRQ Publication No. 13(14)-EHC130-EF. Agency for Healthcare Research and Quality, Rockville, MD2013
- SRs in health care: assessing the quality of controlled clinical trials.BMJ. 2001; 323: 42-46
Chapter 14: Completing ‘Summary of findings’ tables and grading the certainty of the evidence. Available at: https://training.cochrane.org/handbook/current/chapter-14#section-14-1-5. Accessed February 5, 2022.
- Rating the certainty in evidence in the absence of a single estimate of effect.Evid Based Med. 2017; 22: 85-87
- Improving GRADE evidence tables part 1: a randomized trial shows improved understanding of content in summary of findings tables with a new format.J Clin Epidemiol. 2016; 74: 7-18
- Improving GRADE evidence tables part 2: a systematic survey of explanatory notes shows more guidance is needed.J Clin Epidemiol. 2016; 74: 19-27
Higgins JPT, Thomas J, Chandler J, et al., editors. Cochrane Handbook for SRs of Interventions version 62 (updated February 2021) Chapter 24: including non-randomized studies on intervention effects | Cochrane Training. Available at: https://trainingcochraneorg/handbook/current/chapter-24. Accessed January 25, 2022.
- Framework for the synthesis of non-randomised studies and randomised controlled trials: a guidance on conducting a SR and meta-analysis for healthcare decision making.BMJ Evid Based Med. 2022; 27: 109-119
- Majority of SRs published in high-impact journals neglected to register the protocols: a meta-epidemiological study.J Clin Epidemiol. 2017; 84: 54-60
- SR adherence to methodological or reporting quality.Syst Rev. 2017; 6: 131
- Examining the quality of evidence to support the effectiveness of interventions: an analysis of SRs.BMJ Open. 2016; 6e011051
- High quality of evidence is uncommon in Cochrane SRs in anaesthesia, critical care and emergency medicine.Eur J Anaesthesiol. 2017; 34: 808-813
- Use of the GRADE approach in health policymaking and evaluation: a scoping review of nutrition and physical activity policies.Implement Sci. 2020; 15: 37
- Implementing the 27 PRISMA 2020 Statement items for SRs in the sport and exercise medicine, musculoskeletal rehabilitation and sports science fields: the PERSiST (implementing Prisma in Exercise, Rehabilitation, Sport medicine and SporTs science) guidance.Br J Sports Med. 2022; 56: 175-195
- Improving GRADE evidence tables part 3: detailed guidance for explanatory footnotes supports creating and understanding GRADE certainty in the evidence judgments.J Clin Epidemiol. 2016; 74: 28-39
- Guidelines rarely used GRADE and applied methods inconsistently: a methodological study of Australian guidelines.J Clin Epidemiol. 2021; 130: 125-134
- The quality of the evidence according to GRADE is predominantly low or very low in oral health SRs.PLoS One. 2015; 10e0131644
Higgins JPT, Thomas J, Chandler J, et al., editors. Cochrane Handbook for SRs of Interventions version 6.2. Accessed January 20, 2022.
Article info
Publication history
Published online: September 23, 2022
Accepted:
September 7,
2022
Received in revised form:
August 6,
2022
Received:
March 14,
2022
Footnotes
The study was supported by the Italian Ministry of Health “Linea 2—Studi metodologici in ortopedia e riabilitazione”—L2085. The funding sources had no controlling role in the study design, data collection, analysis, interpretation, or report writing. C.L.’s postdoctoral fellowship is supported by a CIHR project grant (2021-2024).
Disclosures: none.
Identification
Copyright
© 2022 by the American Congress of Rehabilitation Medicine.
