Advertisement

One-Third of Systematic Reviews in Rehabilitation Applied the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) System to Evaluate Certainty of Evidence: A Meta-Research Study

  • Author Footnotes
    1 Silvia Gianola and Silvia Bargeri are co-first authors.
    Silvia Gianola
    Correspondence
    Corresponding author Silvia Gianola, IRCCS Istituto Ortopedico Galeazzi, Via R.Galeazzi 4, 20162 Milano, Italy.
    Footnotes
    1 Silvia Gianola and Silvia Bargeri are co-first authors.
    Affiliations
    IRCCS Istituto Ortopedico Galeazzi, Unit of Clinical Epidemiology, Milan, Italy
    Search for articles by this author
  • Author Footnotes
    1 Silvia Gianola and Silvia Bargeri are co-first authors.
    Silvia Bargeri
    Footnotes
    1 Silvia Gianola and Silvia Bargeri are co-first authors.
    Affiliations
    IRCCS Istituto Ortopedico Galeazzi, Unit of Clinical Epidemiology, Milan, Italy
    Search for articles by this author
  • Giulia Nembrini
    Affiliations
    Unità Operativa di Neuropsichiatria Infanzia e Adolescenza (UONPIA), ASST Pavia, Italy
    Search for articles by this author
  • Arianna Varvello
    Affiliations
    Vita-Salute San Raffaele University, Milan, Italy
    Search for articles by this author
  • Author Footnotes
    † Carole Lunny and Greta Castellini are contributed equally to this work.
    Carole Lunny
    Footnotes
    † Carole Lunny and Greta Castellini are contributed equally to this work.
    Affiliations
    Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto, Cochrane Hypertension Review Group, the Therapeutics Initiative, University of British Columbia, Canada
    Search for articles by this author
  • Author Footnotes
    † Carole Lunny and Greta Castellini are contributed equally to this work.
    Greta Castellini
    Footnotes
    † Carole Lunny and Greta Castellini are contributed equally to this work.
    Affiliations
    IRCCS Istituto Ortopedico Galeazzi, Unit of Clinical Epidemiology, Milan, Italy
    Search for articles by this author
  • Author Footnotes
    1 Silvia Gianola and Silvia Bargeri are co-first authors.
    † Carole Lunny and Greta Castellini are contributed equally to this work.
Published:September 23, 2022DOI:https://doi.org/10.1016/j.apmr.2022.09.005

      Highlights

      • One-third of systematic reviews in rehabilitation assess the certainty of evidence
      • The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) was the most common approach used
      • High uptake of approaches such as GRADE is recommended

      Abstract

      Objective

      To determine how many systematic reviews (SRs) of the literature in rehabilitation assess the certainty of evidence (CoE) and how many apply the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system to do this.

      Data Sources

      For this meta-research study, we searched PubMed and Cochrane Database of Systematic Reviews databases for SRs on rehabilitation published in 2020.

      Study Selection and Data Extraction

      Two reviewers independently selected the SRs and extracted the data. Reporting characteristics and appropriate use of the GRADE system were assessed.

      Data Synthesis

      The search retrieved 827 records: 29% (239/827) SRs evaluated CoE, 68% (163/239) of which applied the GRADE system. GRADE was used by SRs of randomized controlled trials (RCTs, 88%; 144/163), non-randomized intervention studies (NRIS, 2%; 3/163), and both RCT and NRIS (10%; 16/163). In the latter case, a separate GRADE assessment according to the study design was not provided in 75% (12/16). The reasons for GRADE judgment were reported in 82% (134/163) of SRs.

      Conclusions

      One-third of SRs in rehabilitation assessed CoE with the GRADE system. GRADE assessment was presented transparently by most SRs. Journal editors and funders should encourage the uptake of the GRADE system when considering SRs in rehabilitation for publication. The authors should pre-define GRADE assessment in a registered and/or published protocol.

      Keywords

      List of abbreviations:

      CoE (certainty of evidence), GRADE (Grading of Recommendations Assessment, Development, and Evaluation), JIF (journal impact factor), NRIS (non-randomized intervention studies), OSF (Open Science Framework), PERSiST (implementing Prisma in Exercise, Rehabilitation, Sport medicine and SporTs science), PRISMA (Preferred Reporting Items for SRs and Meta-analyses), RCT (randomized controlled trials), ROBINS-I (Risk Of Bias In Non-Randomized Studies of Interventions), SIGN (Scottish Intercollegiate Guidelines Network), SoF (summary of findings)
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Archives of Physical Medicine and Rehabilitation
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Moher D
        • Cook DJ
        • Eastwood S
        • et al.
        Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses.
        Lancet. 1999; 354: 1896-1900
        • Kirmayr M
        • Quilodrán C
        • Valente B
        • et al.
        The GRADE approach, Part 1: how to assess the certainty of the evidence.
        Medwave. 2021; 21: e8109
        • Balshem H
        • Helfand M
        • Schunemann HJ
        • et al.
        GRADE guidelines: 3. Rating the quality of evidence.
        J Clin Epidemiol. 2011; 64: 401-406
        • Movsisyan A
        • Dennis J
        • Rehfuess E
        • et al.
        Rating the quality of a body of evidence on the effectiveness of health and social interventions: a SR and mapping of evidence domains.
        Res Synth Methods. 2018; 9: 224-242
        • Guyatt GH
        • Oxman AD
        • Kunz R
        • et al.
        What is "quality of evidence" and why is it important to clinicians?.
        BMJ. 2008; 336: 995-998
        • Guyatt G
        • Oxman AD
        • Akl EA
        • et al.
        GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables.
        J Clin Epidemiol. 2011; 64: 383-394
        • Guyatt GH
        • Oxman AD
        • Vist GE
        • et al.
        GRADE Working Group. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations.
        BMJ. 2008; 336: 924-926
        • Guyatt GH
        • Oxman AD
        • Vist G
        • et al.
        GRADE guidelines: 4. Rating the quality of evidence–study limitations (risk of bias).
        J Clin Epidemiol. 2011; 64: 407-415
        • Guyatt GH
        • Oxman AD
        • Sultan S
        • et al.
        GRADE guidelines: 9. Rating up the quality of evidence.
        J Clin Epidemiol. 2011; 64: 1311-1316
        • Langer G
        • Meerpohl JJ
        • Perleth M
        • et al.
        [GRADE guidelines: 1. Introduction - GRADE evidence profiles and summary of findings tables].
        Z Evid Fortbild Qual Gesundhwes. 2012; 106: 357-368
        • Schunemann HJ
        • Cuello C
        • Akl EA
        • et al.
        GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence.
        J Clin Epidemiol. 2019; 111: 105-114
      1. Schünemann H, Brożek J, Guyatt G, Oxman A, editors. GRADE handbook for grading quality of evidence and strength of recommendations. Updated October 2013. The GRADE Working Group, 2013. Available at: guidelinedevelopment.org/handbook.

        • Page MJ
        • Moher D
        • Bossuyt PM
        • et al.
        PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting SRs.
        BMJ. 2021; 372: n160
        • Page MJ
        • McKenzie JE
        • Bossuyt PM
        • et al.
        The PRISMA 2020 statement: an updated guideline for reporting SRs.
        BMJ. 2021; 372: n71
        • Puljak L
        • Makaric ZL
        • Buljan I
        • et al.
        What is a meta-epidemiological study? Analysis of published literature indicated heterogeneous study designs and definitions.
        J Comp Eff Res. 2020; 9: 497-508
        • Puljak L.
        Methodological research: open questions, the need for 'research on research' and its implications for evidence-based health care and reducing research waste.
        Int J Evid Based Healthc. 2019; 17: 145-146
        • Lawson DO
        • Puljak L
        • Pieper D
        • et al.
        Reporting of methodological studies in health research: a protocol for the development of the MethodologIcal STudy reportIng Checklist (MISTIC).
        BMJ Open. 2020; 10e040478
        • Murad MH
        • Wang Z.
        Guidelines for reporting meta-epidemiological methodology research.
        Evid Based Med. 2017; 22: 139-142
        • Gough D
        • Thomas J
        • Oliver S.
        Clarifying differences between review designs and methods.
        Syst Rev. 2012; 1: 28
        • McGowan J
        • Sampson M
        • Salzwedel DM
        • et al.
        PRESS peer review of electronic search strategies: 2015 guideline statement.
        J Clin Epidemiol. 2016; 75: 40-46
        • PRESS – Peer Review of Electronic Search Strategies
        2015 Guideline Explanation and Elaboration (PRESS E&E).
        CADTH, Ottawa2016
        • EndNote [program]
        EndNote X9 version.
        Clarivate, Philadelphia, PA2013
        • Ouzzani M
        • Hammady H
        • Fedorowicz Z
        • et al.
        Rayyan-a web and mobile app for SRs.
        Syst Rev. 2016; 5: 210
        • Shea BJ
        • Reeves BC
        • Wells G
        • et al.
        AMSTAR 2: a critical appraisal tool for SRs that include randomised or non-randomised studies of healthcare interventions, or both.
        BMJ. 2017; 358: j4008
        • Bun RS
        • Scheer J
        • Guillo S
        • et al.
        Meta-analyses frequently pooled different study types together: a meta-epidemiological study.
        J Clin Epidemiol. 2020; 118: 18-28
        • Shrier I.
        Structural approach to bias in meta-analyses.
        Res Synth Methods. 2011; 2: 223-237
        • Alsan M
        • Finkelstein AN.
        Beyond causality: additional benefits of randomized controlled trials for improving health care delivery.
        Milbank Q. 2021; 99: 864-881
        • Higgins JP
        • Altman DG
        • Gotzsche PC
        • et al.
        The Cochrane Collaboration's tool for assessing risk of bias in randomised trials.
        BMJ. 2011; 343: d5928
        • Valentine JC
        • Thompson SG.
        Issues relating to confounding and meta-analysis when including non-randomized studies in SRs on the effects of interventions.
        Res Synth Methods. 2013; 4: 26-35
        • Stone J
        • Gurunathan U
        • Glass K
        • et al.
        Stratification by quality induced selection bias in a meta-analysis of clinical trials.
        J Clin Epidemiol. 2019; 107: 51-59
        • StataCorp
        Stata Statistical Software: Release 15.
        Stata- Corp LLC, College Station, TX, USA2017
      2. E-bRO. Evidence-Based Guideline Development Manual for Working Group members Original Publication. 2005. Available at: http://www.ha-ring.nl/download/literatuur/EBRO_handl_totaal.pdf. Accessed February 20, 2022.

      3. Howick J, Chalmers I, Glasziou P, et al. “Explanation of the 2011 Oxford Centre for Evidence-Based Medicine (OCEBM) Levels of Evidence (Background Document)”. Oxford Centre for Evidence-Based Medicine. Available at: https://www.cebm.ox.ac.uk/resources/levels-of-evidence/ocebm-levels-of-evidence. Accessed February 20, 2022.

        • Institute JB.
        Joanna Briggs Institute reviewers’ manual.
        The Joanna Briggs Institute, Adelaide2014
        • Lloyd-Smith W.
        Evidence-based practice and occupational therapy.
        Br J Occup Ther. 1997; 60: 474-478
        • Merlin T
        • Weston A
        • Tooher R.
        Extending an evidence hierarchy to include topics other than treatment: revising the Australian 'levels of evidence'.
        BMC Med Res Methodol. 2009; 9: 34
        • Lowe G
        • Twaddle S.
        The Scottish Intercollegiate Guidelines Network (SIGN): an update.
        Scott Med J. 2005; 50: 51-52
      4. U.S. Preventive Services Task Force. U.S. Preventive Services Task Force ratings. Available at: https://www.uspreventiveservicestaskforce.org/Page/Name/us-preventive-services-task-force-ratings. Accessed February 20, 2022.

        • Guyatt GH
        • Oxman AD
        • Vist GE
        • et al.
        GRADE: an emerging consensus on rating quality of evidence and strength of recommendations.
        BMJ. 2008; 336: 924-926
      5. Joanna Briggs Institute. Supporting document for the Joanna Briggs Institute levels of evidence and grades of recommendations. 2014. The Joanna Briggs Institute Levels of Evidence and Grades of Recommendations Working Party. Available at: https://jbi.global/sites/default/files/2019-05/JBI-Levels-of-evidence_2014_0.pdf. Accessed October 26, 2016.

        • Berkman ND
        • Lohr K
        • Ansari M
        • et al.
        Grading the strength of a body of evidence when assessing health care interventions for the effective health care program of the agency for healthcare research and quality: an update.
        AHRQ Publication No. 13(14)-EHC130-EF. Agency for Healthcare Research and Quality, Rockville, MD2013
        • Juni P
        • Altman DG
        • Egger M.
        SRs in health care: assessing the quality of controlled clinical trials.
        BMJ. 2001; 323: 42-46
      6. Chapter 14: Completing ‘Summary of findings’ tables and grading the certainty of the evidence. Available at: https://training.cochrane.org/handbook/current/chapter-14#section-14-1-5. Accessed February 5, 2022.

        • Murad MH
        • Mustafa RA
        • Schunemann HJ
        • et al.
        Rating the certainty in evidence in the absence of a single estimate of effect.
        Evid Based Med. 2017; 22: 85-87
        • Carrasco-Labra A
        • Brignardello-Petersen R
        • Santesso N
        • et al.
        Improving GRADE evidence tables part 1: a randomized trial shows improved understanding of content in summary of findings tables with a new format.
        J Clin Epidemiol. 2016; 74: 7-18
        • Langendam M
        • Carrasco-Labra A
        • Santesso N
        • et al.
        Improving GRADE evidence tables part 2: a systematic survey of explanatory notes shows more guidance is needed.
        J Clin Epidemiol. 2016; 74: 19-27
      7. Higgins JPT, Thomas J, Chandler J, et al., editors. Cochrane Handbook for SRs of Interventions version 62 (updated February 2021) Chapter 24: including non-randomized studies on intervention effects | Cochrane Training. Available at: https://trainingcochraneorg/handbook/current/chapter-24. Accessed January 25, 2022.

        • Sarri G
        • Patorno E
        • Yuan H
        • et al.
        Framework for the synthesis of non-randomised studies and randomised controlled trials: a guidance on conducting a SR and meta-analysis for healthcare decision making.
        BMJ Evid Based Med. 2022; 27: 109-119
        • Tsujimoto Y
        • Tsujimoto H
        • Kataoka Y
        • et al.
        Majority of SRs published in high-impact journals neglected to register the protocols: a meta-epidemiological study.
        J Clin Epidemiol. 2017; 84: 54-60
        • Pussegoda K
        • Turner L
        • Garritty C
        • et al.
        SR adherence to methodological or reporting quality.
        Syst Rev. 2017; 6: 131
        • Kane RL
        • Butler M
        • Ng W.
        Examining the quality of evidence to support the effectiveness of interventions: an analysis of SRs.
        BMJ Open. 2016; 6e011051
        • Conway A
        • Conway Z
        • Soalheira K
        • et al.
        High quality of evidence is uncommon in Cochrane SRs in anaesthesia, critical care and emergency medicine.
        Eur J Anaesthesiol. 2017; 34: 808-813
        • Zähringer J
        • Schwingshackl L
        • Movsisyan A
        • et al.
        Use of the GRADE approach in health policymaking and evaluation: a scoping review of nutrition and physical activity policies.
        Implement Sci. 2020; 15: 37
        • Ardern CL
        • Buttner F
        • Andrade R
        • et al.
        Implementing the 27 PRISMA 2020 Statement items for SRs in the sport and exercise medicine, musculoskeletal rehabilitation and sports science fields: the PERSiST (implementing Prisma in Exercise, Rehabilitation, Sport medicine and SporTs science) guidance.
        Br J Sports Med. 2022; 56: 175-195
        • Santesso N
        • Carrasco-Labra A
        • Langendam M
        • et al.
        Improving GRADE evidence tables part 3: detailed guidance for explanatory footnotes supports creating and understanding GRADE certainty in the evidence judgments.
        J Clin Epidemiol. 2016; 74: 28-39
        • Barker TH
        • Dias M
        • Stern C
        • et al.
        Guidelines rarely used GRADE and applied methods inconsistently: a methodological study of Australian guidelines.
        J Clin Epidemiol. 2021; 130: 125-134
        • Pandis N
        • Fleming PS
        • Worthington H
        • et al.
        The quality of the evidence according to GRADE is predominantly low or very low in oral health SRs.
        PLoS One. 2015; 10e0131644
      8. Higgins JPT, Thomas J, Chandler J, et al., editors. Cochrane Handbook for SRs of Interventions version 6.2. Accessed January 20, 2022.