Advertisement
REVIEW ARTICLE (META-ANALYSIS)| Volume 103, ISSUE 9, P1839-1847, September 2022

Completeness of Reporting Is Suboptimal in Randomized Controlled Trials Published in Rehabilitation Journals, With Trials With Low Risk of Bias Displaying Better Reporting: A Meta-research Study

Open AccessPublished:February 18, 2022DOI:https://doi.org/10.1016/j.apmr.2022.01.156

      Abstract

      Objective

      Primary: To evaluate the completeness of reporting of randomized controlled trials (RCTs) published in rehabilitation journals through the evaluation of the adherence to the Consolidated Standards of Reporting Trials (CONSORT) checklist and investigate the relationship between reporting and risk of bias (ROB). Secondary: To study the association between completeness of reporting and the characteristics of studies and journals.

      Data Sources

      A random sample of 200 RCTs published between 2011 and 2020 in 68 rehabilitation journals indexed under the “rehabilitation” category in the InCites Journal Citation Report.

      Study Selection

      One reviewer evaluated the completeness of reporting operationalized as the adherence to the CONSORT checklist. Two independent reviewers evaluated the ROB using the Cochrane risk-of-bias 2.0 tool.

      Data Extraction

      Overall adherence and adherence to each CONSORT section were calculated. Regression analyses investigated the association between completeness of reporting, ROB, and other characteristics (quartile range, publication modalities, study protocol registration).

      Data Synthesis

      The mean overall CONSORT adherence across studies was 65%. Studies with high ROB have less adherence than those with low ROB (−5.5%; CI, −10.9 to 0.0). There was a 10.2% (% CI, 6.2-14.3) increase in adherence if the RCT protocol was registered. Studies published in first quartile journals displayed an overall adherence of 11.7% (% CI 17.1-6.4) higher than those published in the fourth quartile.

      Conclusions

      Reporting completeness is still suboptimal and is associated with ROB, journal impact ranking, and registration of the study protocol. Trial authors should improve adherence to the CONSORT guideline, and journal editors should adopt new strategies to improve the reporting.

      Keywords

      List of abbreviations:

      CONSORT (Consolidated Standards of Reporting Trials), IF (impact factor), RCT (randomized controlled trial), RG (reporting guideline), ROB (risk of bias), RoB 2 (risk-of-bias tool 2.0)
      Randomized controlled trials (RCTs) are considered the preferable source of evidence to assess the effectiveness of an intervention for a health condition.
      • Altman DG.
      Transparent reporting of trials is essential.
      Evaluating the validity of a trial requires essential information to be included in a study report so that readers can adequately assess the generalizability of the study and decide on whether it is appropriate to apply the findings to their patient population.
      • Rennie D.
      How to report randomized controlled trials: the CONSORT Statement.
      In fact, the purpose of a research report is to communicate the design, the execution, and the findings of a study with precision and accuracy
      • Golub RM
      • Fontanarosa PB.
      Researchers, readers, and reporting guidelines: writing between the lines.
      to be relevant for several stakeholder groups, including researchers, clinicians, policymakers, and patients.
      • Altman DG
      • Moher D.
      • Altman DG
      • Moher D
      • Schultz KF
      • Simera I
      • Wagner E
      Importance of transparent reporting of health research.
      Beyond limiting the applicability of study results, one of the consequences of poor reporting is its possible influence on the risk of bias (ROB) evaluation. Study methods are frequently not described adequately, and results are presented ambiguously, incompletely, or selectively.
      • Altman DG
      • Moher D.
      • Altman DG
      • Moher D
      • Schultz KF
      • Simera I
      • Wagner E
      Importance of transparent reporting of health research.
      The consequence is that the internal validity of a study, in terms of ROB, is hard to judge. Therefore, these studies are difficult to use for decision-making purposes, such as developing clinical guidelines. To overcome these problems, reporting guidelines (RGs) have been developed to support authors in reporting research methods, results, and other sections of a scientific manuscript.
      • Chan L
      • Heinemann AW
      • Roberts J.
      Elevating the quality of disability and rehabilitation research: mandatory use of the reporting guidelines.
      The Consolidated Standards of Reporting Trials (CONSORT) Statement is used worldwide as a reporting guideline focused on RCTs.
      • Altman DG
      • Simera I.
      A history of the evolution of guidelines for reporting medical research: the long road to the EQUATOR Network.
      The first statement was published in 1996,
      • Begg C
      • Cho M
      • Eastwood S
      • et al.
      Improving the quality of reporting of randomized controlled trials. The CONSORT Statement.
      was updated in 2010,
      • Schulz KF
      • Altman DG
      • Moher D
      CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials.
      and currently consists of 25 key items and 11 subitems (for a total of 37 items) that guide the reporting of an RCT. Assessing published RCTs for their completeness of reporting (ie, adherence to the CONSORT checklists) is essential for drafting further publication policies and minimizing the risk of publication bias.
      • Turner L
      • Shamseer L
      • Altman DG
      • et al.
      Consolidated Standards of Reporting Trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals.
      Several studies have assessed adherence to the CONSORT checklist in RCTs in the medical field.
      • Mozetic V
      • Leonel L
      • Leite Pacheco R
      • et al.
      Reporting quality and adherence of randomized controlled trials about statins and/or fibrates for diabetic retinopathy to the CONSORT checklist.
      ,
      • Lai R
      • Chu R
      • Fraumeni M
      • Thabane L.
      Quality of randomized controlled trials reporting in the primary treatment of brain tumors.
      These studies have shown poor overall adherence to the CONSORT checklist items, especially for the items of the methods section. A previous study
      • Innocenti T
      • Salvioli S
      • Giagio S
      • Feller D
      • Cartabellotta N
      • Chiarotto A.
      Declaration of use and appropriate use of reporting guidelines in high-impact rehabilitation journals is limited: a meta-research study.
      confirmed that most of the authors (∼88%) publishing RCTs in high impact rehabilitation journals did not mention RG use and that 58% of those who declared using a RG did not do it appropriately. This occurred despite many rehabilitation journals endorsing RGs during the submission process.
      • Innocenti T
      • Salvioli S
      • Giagio S
      • Feller D
      • Cartabellotta N
      • Chiarotto A.
      Declaration of use and appropriate use of reporting guidelines in high-impact rehabilitation journals is limited: a meta-research study.
      In fact, in 2014, a total of 28 rehabilitation journals simultaneously published an editorial highlighting the need of using RGs to ensure the quality and completeness of studies in the field. To our knowledge, adherence to the CONSORT checklist and the relationship between the completeness of reporting and ROB of RCTs published in rehabilitation journals have not been systematically evaluated.
      The primary objectives of this meta-research study were (1) to evaluate the completeness of reporting of RCTs published in rehabilitation journals through the evaluation of the adherence to the CONSORT checklist and (2) to investigate the relationship between the completeness of reporting and the ROB assessed with the Cochrane risk-of-bias tool 2.0 (RoB 2) tool, which is the most updated Cochrane tool to assess the ROB in RCTs.
      • Sterne JAC
      • Savovic J
      • Page MJ
      • et al.
      RoB 2: a revised tool for assessing risk of bias in randomised trials.
      The secondary objective was to study the association between the completeness of reporting and the characteristics of studies and journals.

      Methods

      A cross-sectional analysis was conducted in a random sample of 200 RCTs published between 2011 and 2020 in the 68 journals indexed under the “rehabilitation” category in InCites Journal Citation Report.

      Clarivate. Journal citation report. Availabile at https://jcr.clarivate.com. Accessed March 28, 2020.

      This sampling method was chosen to include all journals in the rehabilitation field. The 68 rehabilitation journals indexed in InCites are reported as an annex (supplemental table S1, available online only at http://www.archives-pmr.org/). The project protocol was publicly posted before data extraction on the medRxiv preprint server.

      Innocenti T, Giagio S, Salvioli S, et al. Adherence to the CONSORT Statement and risk of bias assessment in randomized controlled trials in rehabilitation journals: a protocol for a meta-research study. medRxiv 2021:2021.01.21.21250256.

      Given that a specific reporting checklist for meta-research studies is currently under development,
      • Lawson DO
      • Puljak L
      • Pieper D
      • et al.
      Reporting of methodological studies in health research: a protocol for the development of the Methodological Study Reporting Checklist (MISTIC).
      we followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses 2020 checklist
      • Page MJ
      • McKenzie JE
      • Bossuyt PM
      • et al.
      The PRISMA 2020 Statement: an updated guideline for reporting systematic reviews.
      to report this article.

      Study selection criteria

      RCTs with a parallel-group design published from 2011-2020 as full-text scientific articles in the rehabilitation journals as mentioned earlier were included. The following were excluded: observational or cohort studies, interim analyses, economic analyses of RCTs, RCT protocols, quasi-experimental design, posttrial follow-up studies, subgroup and secondary analyses of previously reported RCTs, RCTs with crossover design, pilot feasibility RCTs, n-of-1 trials, cluster trials, editorials, letters, and news reports.

      Study selection process

      Journal tags for the journals were identified in MEDLINE, and a detailed search strategy was created to find all RCTs published from 2011-2020 in this database (supplemental table S2, available online only at http://www.archives-pmr.org/). The search was performed only in MEDLINE because all rehabilitation journals are indexed in this database. Titles and abstracts of the selected articles were screened for eligibility in a blinded process by 2 independent reviewers who selected potential articles based on inclusion criteria. A third reviewer resolved any disagreement. After this process, 200 reports (our study sample) were randomly selected by an independent author using a computerized random sequence generator.

      Random.org. Random sequence generator. Availabile at https://www.random.org/sequences/. Accessed February 15, 2021.

      Randomization was stratified by publication date and journal ranking (quartile range: Q1, Q2, Q3, and Q4, based on InCites

      Clarivate. Journal citation report. Availabile at https://jcr.clarivate.com. Accessed March 28, 2020.

      ) to include an equal number of studies from 2011-2015 (n=25 for each quartile range) and from 2016-2020 (n=25 for each quartile range). The study selection process is summarized in fig 1.
      Fig 1
      Fig 1Flowchart of the study selection process.

      Data extraction

      Full texts were stored in EndNote X7.a A data extraction form was used and the following data were extracted: first author, publication year, journal characteristics (open access vs hybrid), country, rehabilitation field, protocol registration (yes/no), completeness of the reporting (see below), and ROB with the Cochrane RoB 2 tool (see below).

      Methods for extracting data and assessing the completeness of reporting

      The completeness of reporting was calculated as adherence to each item on the CONSORT checklist,
      • Schulz KF
      • Altman DG
      • Moher D
      CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials.
      and it was assessed using a 25-item checklist. Preliminarily, interrater agreement was evaluated in a sample of 30 RCTs (15% of the entire sample), which were scored independently by 2 authors with postgraduate training in clinical epidemiology and critical appraisal. This method was chosen to ensure reproducibility and to align the appraisers. However, the CONSORT checklist is not an assessment checklist; therefore, it does not need to be performed independently by 2 authors. We calculated the Cohen kappa for the couple of raters to measure the interrater agreement. The agreement was classified as suggested by McHugh
      • McHugh ML.
      Interrater reliability: the kappa statistic.
      : values <0 indicate no agreement, 0-0.20 none to slight, 0.21-0.40 fair, 0.41-0.60 moderate, 0.61-0.80 substantial, and 0.81-1 almost perfect agreement. In this sample, the agreement was almost perfect (κ=0.825; P<.001).
      Following the explanation and elaboration statements of the CONSORT checklist, each item was marked with “1” if it was well described, with “0” if incomplete or missing, and with “NA” if not applicable. An item was defined as “NA” when its authors did not describe it, but it was clear (by reading the protocol and/or the full text) that such information was justifiably missing (eg, if authors did not perform any additional analyses, such as a priori subgroup analysis because they did not plan them in the protocol, item 12b was not applicable). Total adherence to each item and overall adherence to the CONSORT checklist for each study were calculated (from 0%-100% with 0% representing 0 adherence and 100% full adherence), weighting for the number of applicable items. Given the aim of this study to investigate the completeness of reporting, the authors of the studies included were not contacted for information omitted from articles.

      Methods for extracting data and assessing the risk of bias

      The RoB 2 tool
      • Sterne JAC
      • Savovic J
      • Page MJ
      • et al.
      RoB 2: a revised tool for assessing risk of bias in randomised trials.
      was used to assess the ROB in the studies included. Because this assessment is required for each outcome measure considered critical or important, the primary outcome of each RCT was considered for evaluation in this study. For each RCT, a ROB level was assigned for each of the 5 domains (bias arising from the randomization process, bias because of deviations from intended interventions, bias because of missing outcome data, bias in the measurement of the outcome, bias in the selection of the reported result), as suggested by the Cochrane Collaboration tool.
      • Sterne JAC
      • Savovic J
      • Page MJ
      • et al.
      RoB 2: a revised tool for assessing risk of bias in randomised trials.
      Possible ROB level was “high,” “low,” or “some concerns.” An overall ROB score for each study was assigned as suggested by the tool's instructions. The results were graphically summarized through the ROB graph obtained with the ROBVIS Tool.
      • McGuinness LA
      • Higgins JPT.
      Risk-of-bias visualization (robvis): an R package and Shiny web app for visualizing risk-of-bias assessments.
      Two reviewers performed the ROB assessment for each RCT independently; a third reviewer resolved the disagreement when necessary.

      Data analysis

      The primary analysis addressed (1) the completeness of reporting in each study (see above). Overall adherence to the CONSORT checklist was calculated (in percentage) as the total number of items described and reported out of the total number of applicable items. (2) The adherence to each item and each section of the CONSORT checklist in all studies, calculated as (for each item; in percentage) the number of times that 1 item is described and reported out of the total number of studies in which the item could potentially appear in. (3) The relationship between the completeness of reporting for each study (see above) and studies with low or high ROB. We hypothesized that studies with domains with a low ROB display better reporting compared with those with a high ROB. The relationship was investigated both through descriptive statistics and with multivariable linear regression analysis between the adherence to the CONSORT checklist (both overall adherence and adherence to each checklist section) as dependent variable(s) and the ROB (for each RoB 2 domain and overall ROB) as independent variables.
      A multivariable linear regression analysis was performed as a secondary analysis to assess the association between the adherence to the CONSORT checklist (both overall adherence and adherence to each section) as dependent variable(s) and the following characteristics as independent variables: publication year; journal ranking (quartile range: Q1, Q2, Q3, Q4); publication options (Open Access vs Hybrid); and study protocol registration (yes/no).
      The assumptions of linearity, homoscedasticity, independence of observations, and normality were checked for each linear regression model.

      Results

      The journals and studies' characteristics are shown in supplemental table S3 (available online only at http://www.archives-pmr.org/), and their references are listed in supplementary table S4 (available online only at http://www.archives-pmr.org/). For all regression analyses the assumptions were met.

      Adherence to the CONSORT checklist

      Overall, the mean adherence varied across items (table 1). The most reported item was the background (#2a) in 100% of cases; the least reported one was the description of the trial design (#3a) in 11% of cases. Some items were not applicable in most RCTs (see table 1), and this may have influenced the calculation of adherence of some items (eg, for #14b where adherence was 0% and NA=184). More in general, the highest adherence rates were in the introduction section (99%), and the lowest were in the “other information” section (37%) (fig 2). The mean overall adherence across studies was 65%, with great variability among them. The highest adherence was 97%, and the lowest was 25% (fig 3).
      Table 1Mean adherence across each item of the CONSORT checklist in RCTs published in rehabilitation journals (N=200)
      CONSORT ItemsStudies Where Item Was Not Applicable (n)Mean Adherence (%) Calculated in Studies Where Item Was Applicable
      Title and abstract
      1aIdentification as a randomized trial in the title095
      1bStructured summary of trial design, methods, results, and conclusions (for specific guidance see CONSORT for abstracts)099
      Introduction
      Background and objectives2aScientific background and explanation of rationale0100
      2bSpecific objectives or hypotheses098
      Methods
      Trial design3aDescription of trial design (such as parallel, factorial) including allocation ratio011
      3bImportant changes to methods after trial commencement (such as eligibility criteria), with reasons19560
      Participants4aEligibility criteria for participants097
      4bSettings and locations where the data were collected067
      Interventions5The interventions for each group with sufficient details to allow replication, including how and when they were actually administered083
      Outcomes6aCompletely defined prespecified primary and secondary outcome measures, including how and when they were assessed066
      6bAny changes to trial outcomes after the trial commenced, with reasons19343
      Sample size7aHow sample size was determined056
      7bWhen applicable, explanation of any interim analyses and stopping guidelines19625
      Sequence

      generation
      8aMethod used to generate the random allocation sequence057
      8bType of randomization; details of any restriction (such as blocking and block size)036
      Allocation

      concealment mechanism
      9Mechanism used to implement the random allocation sequence (such as sequentially numbered containers), describing any steps taken to conceal the sequence until interventions were assigned836
       Implementation10Who generated the random allocation sequence, who enrolled participants, and who assigned participants to interventions025
      Blinding11aIf done, who was blinded after assignment to interventions (eg, participants, care providers, those assessing outcomes) and how2366
      11bIf relevant, description of the similarity of interventions14283
      Statistical methods12aStatistical methods used to compare groups for primary and secondary outcomes083
      12bMethods for additional analyses, such as subgroup analyses and adjusted analyses17544
      Results
      Participant flow13aFor each group, the numbers of participants who were randomly assigned, received intended treatment, and were analyzed for the primary outcome082
      13bFor each group, losses and exclusions after randomization, together with reasons2470
      Recruitment14aDates defining the periods of recruitment and follow-up050
      14bWhy the trial ended or was stopped1840
      Baseline data15A table showing baseline demographic and clinical characteristics for each group090
      Numbers analyzed16For each group, number of participants (denominator) included in each analysis and whether the analysis was by original assigned groups062
      Outcomes and estimation17aFor each primary and secondary outcome, results for each group, and the estimated effect size and its precision (such as 95% CI)066
      17bFor binary outcomes, presentation of both absolute and relative effect sizes is recommended18875
      Ancillary analyses18Results of any other analyses performed, including subgroup analyses and adjusted analyses, distinguishing prespecified from exploratory17757
      Harms19All important harms or unintended effects in each group (for specific guidance see CONSORT for harms)5734
      Discussion
      Limitations20Trial limitations, addressing sources of potential bias, imprecision, and, if relevant, multiplicity of analyses082
      Generalizability21Generalizability (external validity, applicability) of the trial findings067
      Interpretation22Interpretation consistent with results, balancing benefits and harms, and considering other relevant evidence093
      Other Information
      Registration23Registration number and name of trial registry038
      Protocol24Where the full trial protocol can be accessed, if available2021
      Funding25Sources of funding and other support (such as supply of drugs), role of funders052
      Fig 2
      Fig 2Adherence across CONSORT sections and overall adherence.
      Fig 3
      Fig 3Overall adherence across the single studies included.

      Relationship between completeness of reporting and risk of bias

      Overall, studies with low ROB showed better reporting than those with high ROB (table 2 and supplemental table S5, available online only at http://www.archives-pmr.org/). The results of RoB 2 analysis are reported in supplemental table S6 (available online only at http://www.archives-pmr.org/). The descriptive analysis showed that the reporting of the results section presented the largest differences between studies at high and low ROB, especially in domain 1 (ROB arising from the randomization process), domain 2 (ROB because of deviations from the intended interventions), and domain 5 (ROB in the selection of the reported result). Domain 5 showed that the greatest differences between high and low risk studies occurred in terms of the reporting of results, discussion, other information (see table 2). Regression analyses also confirm these trends (table 3, primary analysis). Studies with high ROB in domain 1 have an overall adherence to the CONSORT checklist 6.86% (CI, −12.34 to −1.39) lower than those with low ROB. In domain 2, studies with high ROB have 6.99% (CI, −12.67 to −1.28) lower adherence than those with low ROB. If we look at the overall ROB, studies with high ROB have 5.46% (CI, −10.91 to −0.02) lower adherence than those with low ROB.
      Table 2Mean section adherence for each domains of RoB 2 tool and difference between studies at low risk and high risk
      ROBD1D2D3D4D5
      LowHighDifferencesLowHighDifferencesLowHighDifferencesLowHighDifferencesLowHighDifferences
      CONSORTAdherence (%)
      Title and abstract99.092.9−6.197.592.3−5.296.296.90.796.5100.03.597.495.0−2.4
      Introduction99.097.6−1.4100.0100.00.098.698.4−0.299.2100.00.898.7100.01.3
      Methods65.746.5−19.265.247.7−17.559.151.2−7.958.555.2−3.361.647.1−14.5
      Results73.148.8−24.376.249.2−27.067.956.3−11.667.655.8−11.874.145.5−28.6
      Discussion84.182.5−1.689.376.9−12.480.082.32.382.870.0−12.882.063.3−18.7
      Other information43.926.6−17.343.529.5−14.037.631.3−6.341.327.0−14.349.920.0−29.9
      Overall Adherence71.655.3−16.372.655.7−16.966.360.0−6.366.860.2−6.670.452.9−17.5
      Abbreviations: D1, ROB arising from the randomization process; D2, ROB because of deviations from the intended interventions; D3, ROB because of missing outcome data; D4, ROB in measurement of the outcome; D5, ROB in selection of the reported result.
      Table 3Multivariable linear regression models for primary and secondary study analysis with association between independent variables and overall adherence to the CONSORT checklist
      VariableUnstandardized β (95% CI)SE βStandardized βP Value
      Primary analysis
      Constant70.73 (67.00 to 73.47)
      Statistically significant.
      1.39
      Statistically significant.
      <.001
      Statistically significant.
      Risk of bias (D1 low vs D1 high)−6.86 (−12.34 to −1.39)
      Statistically significant.
      2.77
      Statistically significant.
      −0.18
      Statistically significant.
      .014
      Statistically significant.
      Risk of bias (D2 low vs D2 high)−6.99 (−12.67 to −1.28)
      Statistically significant.
      2.89
      Statistically significant.
      −0.18
      Statistically significant.
      .017
      Statistically significant.
      Risk of bias (D3 low vs D3 high)1.14 (−4.64 to 6.92)2.930.03.698
      Risk of bias (D4 low vs D4 high)0.23 (−6.55 to 7.02)3.440.01.946
      Risk of bias (D5 low vs D5 high)−9.09 (−18.35 to 0.17)4.70−0.13.054
      Overall risk of bias (low vs high)−5.46 (−10.91 to −0.02)
      Statistically significant.
      2.76
      Statistically significant.
      −0.18
      Statistically significant.
      .049
      Statistically significant.
      R2=0.20
      Secondary analysis
      Constant60.90 (54.35 to 67.45)
      Statistically significant.
      3.32
      Statistically significant.
      <.001
      Statistically significant.
      Quartile (Q1 vs Q4)−11.76 (−17.13 to −6.4)
      Statistically significant.
      2.72
      Statistically significant.
      −0.34
      Statistically significant.
      <.001
      Statistically significant.
      Quartile (Q1 vs Q3)−7.20 (−12.25 to −2.15)
      Statistically significant.
      2.56
      Statistically significant.
      −0.21
      Statistically significant.
      .005
      Statistically significant.
      Quartile (Q1 vs Q2)−4.36 (−9.38 to 0.66)2.54−0.12.088
      Year of publication1.17 (0.5 to 1.84)
      Statistically significant.
      0.34
      Statistically significant.
      0.21
      Statistically significant.
      .001
      Statistically significant.
      Publication options (hybrid vs open access)0.26 (−5.1 to 5.61)2.720.01.925
      Protocol registration (yes vs no)10.25 (6.21 to 14.29)
      Statistically significant.
      2.05
      Statistically significant.
      0.33
      Statistically significant.
      <.001
      Statistically significant.
      R2=0.32
      NOTE. Dependent variable: overall adherence to the CONSORT checklist.
      Abbreviations: D1, ROB arising from the randomization process; D2, ROB because of deviations from the intended interventions; D3, ROB because of missing outcome data; D4, ROB in measurement of the outcome; D5, ROB in selection of the reported result; high, high risk of bias; low, low risk of bias; Overall ROB, overall risk of bias; Q, quartile.
      low asterisk Statistically significant.
      The descriptive statistics on adherence to each item in a study at low and high risk for each ROB domain are detailed in supplementary table S5. Regression coefficients for adherence to the individual CONSORT sections are reported in supplemental table S7 (available online only at http://www.archives-pmr.org/).

      Secondary analysis

      There was a significant relationship between adherence to the CONSORT checklist and protocol registration, publication year, and journal quartile (see table 3, secondary analysis). For protocol registration, there was a 10.21% (% CI, 6.21-14.29) increase in overall adherence if the protocol was registered. Overall adherence increased by 1.17% (% CI, 0.50-1.84) for each subsequent year, and studies published in Q1 have an overall adherence of 11.76% (% CI, 17.13-6.43) higher than Q4.
      As for the relationship between adherence to each CONSORT section and the same independent variables, there was a 9.76% (% CI, 3.98-15.55) increase in adherence with the methods section if the protocol was published, and studies published in Q1 complied 10.18% (% CI, 17.87-2.48) more than those published in Q4 (detailed regression coefficients for adherence to the CONSORT sections are reported in supplemental table S8, available online only at http://www.archives-pmr.org/). There was also a significant correlation between adherence in the results, discussion, other information sections, protocol registration, journal quartile, and publication year.

      Discussion

      This meta-research study evaluated the completeness of reporting in a representative sample of RCTs published in rehabilitation journals, and the existing correlations with ROB and other important characteristics were investigated. Overall, the completeness of reporting was suboptimal, with a mean overall adherence to the CONSORT checklist equal to 65%, with a range from 25%-97% (see fig 3). The results of this meta-research study are consistent with others in the more general biomedical field.
      • Mozetic V
      • Leonel L
      • Leite Pacheco R
      • et al.
      Reporting quality and adherence of randomized controlled trials about statins and/or fibrates for diabetic retinopathy to the CONSORT checklist.
      ,
      • Kim DY
      • Park HS
      • Cho S
      • Yoon HS.
      The quality of reporting randomized controlled trials in the dermatology literature in an era where the CONSORT Statement is a standard.
      ,
      • Guo JW
      • Sward KA
      • Beck SL
      • Staggers N.
      Quality of reporting randomized controlled trials in cancer nursing research.
      For example, like in this study, an overview of reviews
      • Jin Y
      • Sanger N
      • Shams I
      • et al.
      Does the medical literature remain inadequately described despite having reporting guidelines for 21 years? - A systematic review of reviews: an update.
      concludes that, despite having RGs for 21 years, the reporting of RCTs is still suboptimal.
      To our knowledge, this is the first study to compare the completeness of reporting of RCTs to the ROB of these studies in the rehabilitation field. Our results show a relationship between them, although the potential causality in this association remains to be proven. This implies that authors of RCTs should pay more attention when writing their manuscripts because their work will be evaluated based on what they have clearly stated. As Douglas Altman reported, “Readers should not have to infer what was probably done, they should be told explicitly.”
      • Altman DG.
      Better reporting of randomised controlled trials: the CONSORT Statement.
      (p570)
      Studies with low ROB showed better overall reporting than those with high ROB, especially for the RoB 2 domains regarding selection methods (domains 1 and 2). We investigated this relationship through a multivariable regression analysis (see table 3) with all ROB domains and overall ROB score in the same model. We adopted this solution to clearly show how reporting can be influenced by both individual domains and the overall score without multicollinearity.
      For almost a decade, most international scientific organizations have highlighted the importance of publishing research findings and study protocols

      All Trials. The AllTrials campaign. Availabile at: https://www.alltrials.net. Accessed July 18, 2021.

      to ensure research reproducibility and integrity and to avoid publication bias. It is well established that the prospective registration of clinical trials and the endorsement of reporting guidelines may reduce research dissemination bias in clinical research.
      • Song F
      • Parekh S
      • Hooper L
      • et al.
      Dissemination and publication of research findings: an updated review of related biases.
      Our results showed that protocol registration was positively correlated with better reporting, and this may add relevant knowledge to the rehabilitation field because these results emphasize the role of study protocols and the necessity of their registration to improve each aspect of research quality, from the design to the reporting and the ROB.
      The completeness of reporting seems to be also correlated with publication year, and this leads to more room for improvement. This trend could be explained by rehabilitation journals' progressive endorsement of RGs, even if the evidence on this topic is inconclusive.
      • Stevens A
      • Shamseer L
      • Weinstein E
      • et al.
      Relation of completeness of reporting of health research to journals' endorsement of reporting guidelines: systematic review.
      Moreover, the impact factor (IF) of the journals seems to correlate with the completeness of reporting: studies published in journals in the first quartile have an overall adherence of 11.7% higher than those published in the fourth (see table 3). Although IF metrics are not an indication of the quality of the studies published in a journal,
      • Saginur M
      • Fergusson D
      • Zhang T
      • et al.
      Journal impact factor, trial effect size, and methodological quality appear scantly related: a systematic review and meta-analysis.
      our results suggest that reporting is more complete in journals with high IF.
      Journals could take several steps to ensure better adherence to the CONSORT checklist. First, journals requiring RGs could use a standardized process to assess whether an article follows RGs. Unfortunately, recent studies show that most of the strategies adopted by journals to improve CONSORT use (such as requiring authors to follow the RG in the Instructions to Authors) do not appear to be always effective.
      • Blanco D
      • Biggane AM
      • Cobo E
      • et al.
      Are CONSORT checklists submitted by authors adequately reflecting what information is actually reported in published papers?.
      ,
      • Blanco D
      • Altman D
      • Moher D
      • Boutron I
      • Kirkham JJ
      • Cobo E.
      Scoping review on interventions to improve adherence to reporting guidelines in health research.
      This phenomenon probably happens because editors and reviewers do not check the accuracy of the information and because there are no electronic systems that could check the accuracy of reporting.
      • Stevens A
      • Shamseer L
      • Weinstein E
      • et al.
      Relation of completeness of reporting of health research to journals' endorsement of reporting guidelines: systematic review.
      ,
      • Agha RA
      • Barai I
      • Rajmohan S
      • et al.
      Support for reporting guidelines in surgical journals needs improvement: a systematic review.
      Additionally, journals may not thoroughly check reviewers' work, and reviewers are not provided with incentives for using (or not using) RGs during their peer-review, although asking authors to provide information about incomplete or missing items from reporting guidelines could improve the manuscript quality.
      • Cobo E
      • Cortés J
      • Ribera JM
      • et al.
      Effect of using reporting guidelines during peer review on quality of final manuscripts submitted to a biomedical journal: masked randomised trial.
      There is a need for more attention to each step of evidence production in rehabilitation research. Researchers must improve compliance with the reporting standard. They must increase the readability and transparency of their research reports by submitting the CONSORT checklist alongside their manuscripts, as already required by some rehabilitation journals. Given the lack of incentives for peer-reviewers, journal editors should pay more attention to the peer-review process and, if possible, adopt electronic systems (eg, through artificial intelligence) to check the accuracy of reporting in a scientific paper. In the last years, many studies focused on the use of artificial intelligence in the peer-review process, with promising results in reducing the editor's workload and speeding up the review process.
      • Chubb J
      • Cowling P
      • Reed D.
      Speeding up to keep up: exploring the use of AI in the research process.
      ,
      • Mrowinski MJ
      • Fronczak P
      • Fronczak A
      • Ausloos M
      • Nedic O.
      Artificial intelligence in peer review: how can evolutionary computation support journal editors?.
      These systems may run checks during the submission process, detecting and prompting for corrections before submission and explicitly requiring that authors follow RGs indications.

      Study limitations

      To our knowledge, this is the first study that has thoroughly investigated the use of CONSORT in RCTs published in rehabilitation journals. The major strength is the inclusion of all journals indexed under the rehabilitation category in InCites (without IF restrictions). Therefore, we have included a sample that could potentially represent the entire population of rehabilitation journals. In addition, to our knowledge, this is the first study to compare the completeness of reporting with the ROB in the included studies.
      However, our study also has some limitations. We used the CONSORT checklist to assess the completeness of reporting, despite it being a guide for writing and not for evaluating quality. We are aware that despite being in line with previous meta-research studies,
      • Mozetic V
      • Leonel L
      • Leite Pacheco R
      • et al.
      Reporting quality and adherence of randomized controlled trials about statins and/or fibrates for diabetic retinopathy to the CONSORT checklist.
      ,
      • Kim DY
      • Park HS
      • Cho S
      • Yoon HS.
      The quality of reporting randomized controlled trials in the dermatology literature in an era where the CONSORT Statement is a standard.
      ,
      • Guo JW
      • Sward KA
      • Beck SL
      • Staggers N.
      Quality of reporting randomized controlled trials in cancer nursing research.
      this fact may represent a limitation.
      • Logullo P
      • MacCarthy A
      • Kirtley S
      • Collins GS.
      Reporting guideline checklists are not quality evaluation forms: they are guidance for writing.
      To overcome this issue, we have tried to clearly define how we have reached the scores, and we run a preliminary pilot test to increase interassessor agreement, especially on items requiring interpretation (sometimes indicated by phrases like “if relevant” or “if applicable”).

      Conclusions

      Despite having RGs for more than 20 years, the completeness of reporting in RCTs in rehabilitation research is still suboptimal. RCTs with low ROB seem to be positively correlated with better reporting, and the registration of study protocols and higher journal ranking are also related to complete reporting. Researchers should improve compliance with the reporting standards, and journals editors should implement new strategies in the peer-review process to ensure more attention is devoted to reporting issues.

      Supplier

      • a.
        EndNote X7; Thomas Reuters, Philadelphia, PA.

      Appendix. Supplementary materials

      References

        • Altman DG.
        Transparent reporting of trials is essential.
        Am J Gastroenterol. 2013; 108: 1231-1235
        • Rennie D.
        How to report randomized controlled trials: the CONSORT Statement.
        JAMA. 1996; 276: 649
        • Golub RM
        • Fontanarosa PB.
        Researchers, readers, and reporting guidelines: writing between the lines.
        JAMA. 2015; 313: 1625-1626
        • Altman DG
        • Moher D.
        • Altman DG
        • Moher D
        • Schultz KF
        • Simera I
        • Wagner E
        Importance of transparent reporting of health research.
        Guidelines for reporting health research: a user's manual. Wiley, Chichester, West Sussex2014: 3-13
        • Chan L
        • Heinemann AW
        • Roberts J.
        Elevating the quality of disability and rehabilitation research: mandatory use of the reporting guidelines.
        Am J Occup Ther. 2014; 68: 127-129
        • Altman DG
        • Simera I.
        A history of the evolution of guidelines for reporting medical research: the long road to the EQUATOR Network.
        J Royal Soc Med. 2016; 109: 67-77
        • Begg C
        • Cho M
        • Eastwood S
        • et al.
        Improving the quality of reporting of randomized controlled trials. The CONSORT Statement.
        JAMA. 1996; 276: 637-639
        • Schulz KF
        • Altman DG
        • Moher D
        CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials.
        J Pharmacol Pharmacother. 2010; 1: 100-107
        • Turner L
        • Shamseer L
        • Altman DG
        • et al.
        Consolidated Standards of Reporting Trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals.
        Cochrane Database Syst Rev. 2012; 11MR000030
        • Mozetic V
        • Leonel L
        • Leite Pacheco R
        • et al.
        Reporting quality and adherence of randomized controlled trials about statins and/or fibrates for diabetic retinopathy to the CONSORT checklist.
        Trials. 2019; 20: 729
        • Lai R
        • Chu R
        • Fraumeni M
        • Thabane L.
        Quality of randomized controlled trials reporting in the primary treatment of brain tumors.
        J Clin Oncol. 2006; 24: 1136-1144
        • Innocenti T
        • Salvioli S
        • Giagio S
        • Feller D
        • Cartabellotta N
        • Chiarotto A.
        Declaration of use and appropriate use of reporting guidelines in high-impact rehabilitation journals is limited: a meta-research study.
        J Clin Epidemiol. 2021; 131: 43-50
        • Sterne JAC
        • Savovic J
        • Page MJ
        • et al.
        RoB 2: a revised tool for assessing risk of bias in randomised trials.
        BMJ. 2019; 366: l4898
      1. Clarivate. Journal citation report. Availabile at https://jcr.clarivate.com. Accessed March 28, 2020.

      2. Innocenti T, Giagio S, Salvioli S, et al. Adherence to the CONSORT Statement and risk of bias assessment in randomized controlled trials in rehabilitation journals: a protocol for a meta-research study. medRxiv 2021:2021.01.21.21250256.

        • Lawson DO
        • Puljak L
        • Pieper D
        • et al.
        Reporting of methodological studies in health research: a protocol for the development of the Methodological Study Reporting Checklist (MISTIC).
        BMJ Open. 2020; 10e040478
        • Page MJ
        • McKenzie JE
        • Bossuyt PM
        • et al.
        The PRISMA 2020 Statement: an updated guideline for reporting systematic reviews.
        BMJ. 2021; 372: n71
      3. Random.org. Random sequence generator. Availabile at https://www.random.org/sequences/. Accessed February 15, 2021.

        • McHugh ML.
        Interrater reliability: the kappa statistic.
        Biochem Med (Zagreb). 2012; 22: 276-282
        • McGuinness LA
        • Higgins JPT.
        Risk-of-bias visualization (robvis): an R package and Shiny web app for visualizing risk-of-bias assessments.
        Res Synth Methods. 2021; 12: 55-61
        • Kim DY
        • Park HS
        • Cho S
        • Yoon HS.
        The quality of reporting randomized controlled trials in the dermatology literature in an era where the CONSORT Statement is a standard.
        Br J Dermatol. 2019; 180: 1361-1367
        • Guo JW
        • Sward KA
        • Beck SL
        • Staggers N.
        Quality of reporting randomized controlled trials in cancer nursing research.
        Nurs Res. 2014; 63: 26-35
        • Jin Y
        • Sanger N
        • Shams I
        • et al.
        Does the medical literature remain inadequately described despite having reporting guidelines for 21 years? - A systematic review of reviews: an update.
        J Multidiscip Healthc. 2018; 11: 495-510
        • Altman DG.
        Better reporting of randomised controlled trials: the CONSORT Statement.
        BMJ. 1996; 313: 570-571
      4. All Trials. The AllTrials campaign. Availabile at: https://www.alltrials.net. Accessed July 18, 2021.

        • Song F
        • Parekh S
        • Hooper L
        • et al.
        Dissemination and publication of research findings: an updated review of related biases.
        Health Technol Assess. 2010; 14: 1-220
        • Stevens A
        • Shamseer L
        • Weinstein E
        • et al.
        Relation of completeness of reporting of health research to journals' endorsement of reporting guidelines: systematic review.
        BMJ. 2014; 348: g3804
        • Saginur M
        • Fergusson D
        • Zhang T
        • et al.
        Journal impact factor, trial effect size, and methodological quality appear scantly related: a systematic review and meta-analysis.
        Syst Rev. 2020; 9: 53
        • Blanco D
        • Biggane AM
        • Cobo E
        • et al.
        Are CONSORT checklists submitted by authors adequately reflecting what information is actually reported in published papers?.
        Trials. 2018; 19: 80
        • Blanco D
        • Altman D
        • Moher D
        • Boutron I
        • Kirkham JJ
        • Cobo E.
        Scoping review on interventions to improve adherence to reporting guidelines in health research.
        BMJ Open. 2019; 9e026589
        • Agha RA
        • Barai I
        • Rajmohan S
        • et al.
        Support for reporting guidelines in surgical journals needs improvement: a systematic review.
        Int J Surg. 2017; 45: 14-17
        • Cobo E
        • Cortés J
        • Ribera JM
        • et al.
        Effect of using reporting guidelines during peer review on quality of final manuscripts submitted to a biomedical journal: masked randomised trial.
        BMJ. 2011; 343: d6783
        • Chubb J
        • Cowling P
        • Reed D.
        Speeding up to keep up: exploring the use of AI in the research process.
        AI Soc, 2021 Oct 15 ([Epub ahead of print])
        • Mrowinski MJ
        • Fronczak P
        • Fronczak A
        • Ausloos M
        • Nedic O.
        Artificial intelligence in peer review: how can evolutionary computation support journal editors?.
        PLoS One. 2017; 12e0184711
        • Logullo P
        • MacCarthy A
        • Kirtley S
        • Collins GS.
        Reporting guideline checklists are not quality evaluation forms: they are guidance for writing.
        Health Sci Rep. 2020; 3: e165