Abstract
Johnston MV, Dijkers MP. Toward improved evidence standards and methods for rehabilitation:
recommendations and challenges.
Interventions and programs for people with disability should be based on the best—the
most discriminating and rigorous—methods of systematic review and knowledge translation
possible. Extant systems for systematic review and practice recommendations have excellent
features but severe difficulties are encountered when attempting to apply them to
disability and rehabilitation. This article identifies issues in evidence synthesis
and linked practice recommendations and describes both new and long-tested methods
to address them. Evidence synthesis in disability and rehabilitation can be improved
by: explicating criteria for evaluating nonrandomized evidence, including the regression
discontinuity, interrupted time series, and single-subject designs, as well as state-of-the-art
methods of analysis of observational studies; greater use of meta-analysis; considering
effect size, direction of biases, and dose-response relationships; employing more
discriminating methods of evaluating flaws in masking, considering also measurement
reliability and objectivity; considering overall biases and conflicts of interest;
increased attention to composition of review panels; and greater transparency in reporting
of the bases of reviewers' judgments. Review methods need to be developed for assistive
technology and for measurement procedures. Application to practice can be improved
by attention to treatment alternatives, explicit evaluation of generalizability, synthesizing
clinical experience as a source of evidence, and a focus on the best—rather than the
ideally most-rigorous—evidence. Study outcomes should be measured and reviewed in
terms meaningful to persons served. In sum, methods are available to improve evidence
synthesis and the application of resulting knowledge. We recommend that these methods
be employed.
Key Words
List of Abbreviations:
AAN (American Academy of Neurology), ACRM (American Congress of Rehabilitation Medicine), AT (assistive technology), D&R (disability and rehabilitation), EBM (evidence-based medicine), EBP (evidence-based practice), GRADE (Grading of Recommendations Assessment, Development and Evaluation), IES (Institute of Education Sciences), PEDro (Physiotherapy Evidence Database), RCT (randomized controlled trial), RDD (regression discontinuity design), SR (systematic review), SSD (single subject/case design), TSRD (time series research design)To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Archives of Physical Medicine and RehabilitationAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- The challenge of evidence in disability and rehabilitation research and practice: a position paper.National Center for Dissemination of Disability Research, Southwest Educational Development Laboratory, Austin2009
- The value of “traditional” reviews in the era of systematic reviewing.Am J Phys Med Rehabil. 2009; 88: 423-430
- Knowing what works in health care: a roadmap for the nation.in: Eden J. Wheatley B. McNeil B. Sox H. Committee on reviewing evidence to identify highly effective clinical services. National Academies Pr, Washington (DC)2008
- Systems for grading the quality of evidence and the strength of recommendations I: critical appraisal of existing approaches The GRADE Working Group.BMC Health Serv Res. 2004; 4: 38
- American Academy of Neurology clinical practice guideline process manual.American Academy of Neurology, St. Paul2004
- Higgins J.P.T. Green S. Cochrane handbook for systematic reviews of interventions. Cochrane Collaboration, Chichester2009
- GRADE: an emerging consensus on rating quality of evidence and strength of recommendations.BMJ. 2008; 336: 924-926
- Systems to rate the strength of scientific evidence.Agency for Healthcare Research and Quality, Rockville2002 (AHRQ Publication No 02-E016)
- U.S. Preventive Services Task Force Procedure Manual.(AHRQ Publication No. 08-05118-EF, July 2008) U.S. Government, Rockville2008 (Accessed March 8, 2012)
- Finding what works in health care: standards for systematic reviews.(editors) Institute of Medicine, National Academies Pr, Washington (DC)2011
- Graham R. Mancher M. Wolman D.M. Greenfield S. Steinberg E. Clinical practice guidelines we can trust. Institute of Medicine, National Academies Pr, Washington (DC)2011
- Interpersonal expectations: effects of the experimenter's hypothesis.in: Rosnow R.L. Rosenthal R. People studying people: artifacts and ethics in behavioral research. W.H. Freeman, New York2009: 138-210
- Empirically supported psychological interventions: controversies and evidence.Annu Rev Psychol. 2001; 52: 685-716
- Clinical practice guideline process manual: 2011 edition.American Academy of Neurology, St. Paul2011
- Incorporating considerations of resources use into grading recommendations.BMJ. 2008; 336: 1170-1173
- Grading quality of evidence and strength of recommendations in clinical practice guidelines.Allergy. 2009; 64: 669-677
- GRADE guidelines: 1.J Clin Epidemiol. 2011; 64: 383-394
- GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology.J Clin Epidemiol. 2011; 64: 380-382
- Evidence-based practice for rehabilitation professionals: concepts and controversies.Arch Phys Med Rehabil. 2012; 93: S164-S176
- Evidence-based medicine: how to practice and teach EBM.Churchill Livingstone, Edinburgh2000
- PEDro: a physiotherapy evidence database.Med Ref Serv Q. 2008; 27: 189-198
- Experimental and quasi-experimental designs for research.R. McNally, Chicago1966
- Experimental and quasi-experimental designs for generalized causal inference.Houghton Mifflin, Boston2002
- Quasi-experimentation: design and analysis issues for field settings.Houghton Mifflin, Boston1979
- Strong quasi-experimental designs for research on the effectiveness of rehabilitation.Am J Phys Med Rehabil. 1995; 74: 383-392
- Regression discontinuity designs: a guide to practice.National Bureau of Economic Research, Cambridge2007
- Standards for regression discontinuity designs.in: Institute for Educational Sciences, What Works Clearinghouse, Department of Education. Institute for Educational Sciences, Washington (DC)June 2010 (editors) (Accessed March 8, 2012)
- Grading quality of evidence and strength of recommendations in clinical practice guidelines part 3 of 3.Allergy. 2011; 66: 588-595
- Users' Guides to the Medical Literature: XXV.JAMA. 2000; 284: 1290-1296
- Foundations of clinical research: applications to practice.3rd ed. Pearson/Prentice Hall, Upper Saddle River2009
- Synthesizing efficacy research in AAC.in: Schlosser R. Lloyd L. Schlodber R.W. The efficacy of augmentative and alternative communication. Academic Pr, San Diego2003: 230-257
- The role of single-subject experimental designs in evidence-based practice times.SEDL, National Center for Dissemination of Rehabilitation Research, Austin, TX2009 (Technical Brief No. 22 2009)
- Rating the methodological quality of single-subject designs and n-of-1 trials: introducing the Singe-Case Experimental Design (SCED) Scale.Neuropsychol Rehabil. 2008; 18: 385-401
- Single-subject research in rehabilitation: a review of studies using AB, withdrawal, multiple baseline, and alternating treatments designs.Arch Phys Med Rehabil. 1997; 78: 1145-1153
- Single subject designs in biomedicine.Springer, Dordrecht2009
- Single subject designs: current methodologies and future directions.Occup Ther J Res. 2010; 30: 4-10
- Single-case designs technical documentation.2010 (What Works Clearinghouse, Institute of Educational Sciences, U.S. Department of Education, 2010) (Accessed February 8, 2012)
- Single-case research methods for the behavioral and health sciences.SAGE, Los Angeles2009
- Single-case research designs: methods for clinical and applied settings.2nd ed. Oxford Univ Pr, New York2011
- Time series analysis with applications in R.2nd ed. Springer, New York2008
- Interrupted time series analysis in clinical research.Ann Pharmacother. 2003; 37: 1110-1116
- Interrupted time series designs in health technology assessment: lessons from two systematic reviews of behavior change strategies.Int J Technol Assess Health Care. 2003; 19: 613-623
- Evidence-based practice: methods to evaluate individual patient improvement.Am J Phys Med Rehabil. 2001; 80: 786-796
- Statistical analysis of stochastic processes in time.Cambridge Univ Pr, Cambridge2004
- Current methods of the US Preventive Services Task Force: a review of the process.Am J Preventive Medicine. 2001; 20 (3 (Suppl): 21-35
- Estimating causal effects: using experimental and observational designs: a think tank white paper.American Educational Research Association, Washington (DC)2007
- Meta-analysis of observational studies in epidemiology: a proposal for reporting.JAMA. 2000; 283: 2008-2012
- The multiple propensity score for analysis of dose-response relationships in drug safety studies.Pharmacoepidemiol Drug Saf. 2001; 10: 105-111
- Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies.BMJ. 2007; 335: 806-808
- Estimating causal effects from large data sets using propensity scores.Ann Intern Med. 1997; 127: 757-763
- Assessment and control for confounding by indication in observational studies.J Am Geriatr Soc. 1999; 47: 749-754
- Statistical methods.Transportation Research Board, National Academy of Sciences, Washington (DC)2008
- Causal effects in clinical and epidemiological studies via potential outcomes: concepts and analytical approaches.Annu Rev Public Health. 2000; 21: 121-145
- Randomized trials and observational studies: still mostly similar results, still crucial differences.J Clin Epidemiol. 2008; 61: 207-208
- Observational studies for intervention assessment.Lancet. 2001; 357: 2141
- Instrumental variables and inverse probability weighting for causal inference from longitudinal observational studies.Stat Methods Med Res. 2004; 13: 17-48
- Assessing observational studies of medical treatments.Emerg Themes Epidemiol. 2005; 2: 8
- Observational studies for intervention assessment.Lancet. 2001; 357: 2141-2142
- Why we need observational studies to evaluate the effectiveness of health care.BMJ. 1996; 312: 1215-1218
- Use of instrumental variables in the presence of heterogeneity and self-selection: an application to treatments of breast cancer patients.Health Econ. 2007; 16: 1133-1157
- Proceeding from observed correlation to causal inference: the use of natural experiments.Perspect Psychol Sci. 2007; 2: 377-395
- Randomisation to protect against selection bias in healthcare trials.Cochrane Database Syst Rev. 2011 Apr 13; (MR000012)
- Observational studies.2nd ed. Springer, New York2010
- Using matching, instrumental variables, and control functions to estimate economic choice models.Rev Econ Stat. 2004; 86: 30-57
- A comparison of observational studies and randomized controlled trials.N Engl J Med. 2000; 342: 1878-1886
- The importance of covariate selection in controlling for selection bias in observational studies.Psychol Methods. 2010; 15: 250-267
- Alternatives to the randomized controlled trial.Am J Public Health. 2008; 98: 1359-1366
- Comparison of logistic regression versus propensity score when the number of events is low and there are multiple confounders.Am J Epidemiol. 2003; 158: 280-287
- Instrumental variables: application and limitations.Epidemiol. 2006; 17: 260-267
- Econometrics in outcomes research: the use of instrumental variables.Annu Rev Pub Health. 1998; 19: 17-34
- The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration.J Clin Epidemiol. 2009; 62: e1-e34
- Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.J Clin Epidemiol. 2009; 62: 1006-1012
- The handbook of research synthesis and meta-analysis.2nd ed. Russell Sage Foundation, New York2009
- Best evidence synthesis: an intelligent alternative to meta-analysis.J Clin Epidemiol. 1995; 48: 9-18
GRADEpro. [Computer program]. Version 3.2 for Windows. Jan Brozek, Andrew Oxman, Holger Schünemann, 2008.
- Best evidence synthesis: an intelligent alternative to meta-analysis.J Clin Epidemiol. 1995; 48: 9-18
- Systematically assessing and improving the quality and outcomes of medical rehabilitation programs.in: Frontera W.R. DeLisa J.A. DeLisa's physical medicine & rehabilitation: principles and practice. 5th ed. Wolters Kluwer/Lippincott Williams & Wilkins Health, Philadelphia2010: 325-355
- Treatment strength in rehabilitation.Arch Phys Med Rehabil. 1997; 78: 1298-1304
- Tutorials in clinical research: part IV: recognizing and controlling bias.Laryngoscope. 2002; 112: 23-31
- Applying evidence standards to rehabilitation research: an overview.Am J Phys Med Rehabil. 2006; 85: 292-309
- Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study.BMJ. 2008; 336: 601-605
- Stone A.A. Turkkan J.S. Bachrach C.A. Jobe J.B. Kurtzman H.S. Cain V.S. The science of self-report: implications for research and practice. Lawrence Erlbaum, Mahwah2000
- Internet, mail, and mixed-mode surveys: the tailored design method.3rd ed. Wiley, Hoboken2008
- Faking it: social desirability response bias in self-report research.Austrailan J Adv Nursing. 2008; 25: 40-48
- Rehabilitation outcome measures.Churchill-Livingston, Phildelphia2011
- Those who have the gold make the evidence: how the pharmaceutical industry biases the outcomes of clinical trials of medications.Sci Eng Ethics. 2011 Feb 15; ([Epub ahead of print])
- Pharmaceutical industry sponsorship and research outcome and quality: systematic review.BMJ. 2003; 326: 1167-1170
- Standards for educational and psychological testing.American Educational Research Association, Washington (DC)1999
- Measurement standards for interdisciplinary medical rehabilitation.Arch Phys Med Rehabil. 1992; 73: S3-S23
- International conference on objective measurement: applications of Rasch analysis in health care.Med Care. 2004; 42: 1-6
- Postscript: applications of Rasch analysis in health care.Med Care. 2004; 42: I73-I78
- Criteria for assessing the tools of disability outcomes research.Arch Phys Med Rehabil. 2000; 81: S15-S20
- The uniform postacute assessment tool: systematically evaluating the quality of measurement evidence.Arch Phys Med Rehabil. 2007; 88: 1505-1512
- Towards guidelines for evaluation of measures: an introduction with application to spinal cord injury.J Spinal Cord Med. 2008; 31: 13-26
- Experimental and quasi-experimental designs for research.Houghton Mifflin, Boston1981
- What is “quality of evidence” and why is it important to clinicians?.BMJ. 2008; 336: 995-998
- FORM: an Australian method for formulating and grading recommendations in evidence-based clinical guidelines.BMC Med Res Methodol. 2011; 11: 23
- Fundamentals of clinical trials.4th ed. Springer, New York2010
- Measuring physical disablement: the contextual challenge.Phys Ther. 1994; 74: 443-451
- Extending the CONSORT statement to randomized trials of nonpharmacologic treatment: explanation and elaboration.Ann Intern Med. 2008; 148: 295-309
- Developing and evaluating complex interventions: the new Medical Research Council guidance.BMJ. 2008; 337: 1655
- External validity in research on rehabilitation interventions: issues for knowledge translation.National Center for Dissemination of Disability Research, SEDL, Austin, TX2011 (Accessed February 8, 2012)
- Empirically supported therapies: best practice in professional psychology.Sage, Thousand Oaks1998
- Motivational interviewing in health care; helping patients change behavior.Guilford Pr, New York2008
- The therapeutic alliance: an evidence-based guide to practice.Guilford Pr, New York2010
- Conceptual foundations of occupational therapy practice.4th ed. F.A. Davis Co, Philadelphia2009
- Glossary of terms in The Cochrane Collaboration.(Accessed July 13, 2011)
- Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy.JAMA. 2003; 290: 1624-1632
- Pragmatic trials in primary care.BMC Med Res Methodol. 2007; 7: 16
- A quasi-sampling theory of the generalization of causal relationships.New Directions for Program Evaluation. 1993; 1993: 39-82
- Generalizability theory.Springer, New York2001
- Generalizability theory: a primer.Sage Publications, Newbury Park1991
- Practice based evidence: incorporating clinical heterogeneity and patient-reported outcomes for comparative effectiveness research.Med Care. 2010; 48: S17-S22
- Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science.Implement Sci. 2009; 4: 50
- The development of clinical practice guidelines and guidance statements of the American College of Physicians: summary of methods.Ann Intern Med. 2010; 153: 194-199
- Cognitive behavioural therapy for depression, panic disorder and generalized anxiety disorder: a meta-regression of factors that may predict outcome.Aust N Z J Psychiatry. 2005; 40: 9-19
- What works for whom?.2nd ed. Guilford Pr, New York2005
- Antidepressants versus placebo for depression in primary care.Cochrane Database Syst Rev. 2009 Jul 8; (CD007954)
- Going from evidence to recommendations.BMJ. 2008; 336: 1049-1051
- Crossing the quality chasm: a new health care system for the 21st century.National Academy Pr, Washington (DC)2001
- Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials.BMJ. 2003; 327: 1459-1461
- Asking well-built questions for evidence-based practice in augmentative and alternative communication.J Commun Disord. 2007; 40: 225-238
- Scherer M.J. Assistive technology: matching device and consumer for successful rehabilitation. 1st ed. American Psychological Association, Washington (DC)2002
- Assistive technologies: principles and practice.2nd ed. Mosby, St. Louis2002
- Assistive technology outcomes research: challenges met and yet unmet.Am J Phys Med Rehabil. 2001; 80: 528-535
- Effectiveness of assistive technology and environmental interventions in maintaining independence and reducing home care costs for the frail elderly.Arch Fam Med. 1999; 8: 210-217
- Development and testing of interventions in occupational therapy: towards a new generation of research in occupational therapy.Occup Ther J Res. 2009; 29: 4-13
- Theory as method: small theories of treatment.New Directions for Program Evaluation. 1993; 57: 5-38
- Introduction to statistical mediation analysis.Lawrence Erlbaum Associates, New York2008
- Using treatment theories to refine the designs of brain injury rehabilitation treatment effectiveness studies.J Head Trauma Rehabil. 2006; 21: 99-106
Article info
Footnotes
Presented to the AcademyHealth Annual Research Meeting, June 11, 2011, Seattle, WA. The points in the article have also been discussed in meetings of the American Congress of Rehabilitation Medicine's Clinical Practice Committee and the Task Force on Standards and Methods of Evidence of the National Center for Dissemination of Disability Research.
Supported by the National Institute on Disability and Rehabilitation Research (grant no. H133A060028) and intramural funds of the University of Wisconsin-Milwaukee.
No commercial party having a direct financial interest in the results of the research supporting this article has or will confer a benefit on the authors or on any organization with which the authors are associated.
Reprints are not available from the author.
Identification
Copyright
© 2012 American Congress of Rehabilitation Medicine. Published by Elsevier Inc. All rights reserved.
