Archives of Physical Medicine and Rehabilitation
Volume 86, Issue 12, Supplement , Pages 121-123, December 2005

The Post-Stroke Rehabilitation Outcomes Project

  • Kenneth J. Ottenbacher, PhD

      Affiliations

    • Corresponding Author InformationCorrespondence to Kenneth J. Ottenbacher, PhD, Div of Rehabilitation Sciences, University of Texas Medical Branch, 301 University Blvd, Rt 1137, Galveston, TX 77555-1137

Division of Rehabilitation Sciences and PAHO/WHO Collaborating Center on Aging and Health, University of Texas Medical Branch, Galveston, TX

Article Outline

Abstract 

Ottenbacher KJ. The Post-Stroke Rehabilitation Outcomes Project.

The Post-Stroke Rehabilitation Outcomes Project (PSROP) examined a large sample of patients from multiple facilities receiving inpatient stroke rehabilitation services. This commentary describes strengths and potential limitations of the investigation including selection bias, observation bias, confounds, and interpretation. The PSROP is an important study that will advance our understanding of effective treatment for persons with stroke.

Key Words:  Rehabilitation , Stroke

 

THE POST-STROKE REHABILITATION Outcomes Project (PSROP) is an important study on a major public health problem.1 The World Health Organization projects that stroke will become a leading cause of mortality worldwide in this century.2, 3 More than 50% of those 65 years and older who survive a stroke report hemiparesis at 6 month follow-up. Thirty percent of persons (≥65y) are unable to walk without assistance and 26% are dependent in at least 1 basic activity of daily living 6 months poststroke.1 The social and economic impact of stroke are well described by DeJong et al4 and demonstrate the need for the PSROP.

During the past 2 decades, dramatic improvements have been made in the treatment of stroke and convincing scientific evidence now exists that stroke rehabilitation programs are effective. The evidence on overall efficacy is summarized in Recovery After Stroke5 and other recent publications.6, 7 What we do not know is which patients are the best candidates for the complex array of rehabilitation methods and techniques currently available.6 DeJong refers to this problem as the “black box” of stroke rehabilitation and notes: “the interaction between each stroke survivor, his/her comorbidities, personal behaviors, and coping abilities and all of these health care providers and family members is complex and highly specific—with each and all factors having a possible impact on the patient’s outcome.”4(p2) The PSROP investigators should be congratulated on their efforts to examine systematically the black box of stroke rehabilitation, including the complex interactions among patient characteristics, treatment approaches, and rehabilitation outcomes.

The method they have selected, clinical practice improvement (CPI), is an innovative and powerful approach designed to examine complicated research questions in a real-world setting. The findings are presented as a series of observational “cohort” investigations that represent a descriptive epidemiology of stroke rehabilitation. The strengths of the approach are the relevance to real-world clinical practice, the focus on the care management process, the inclusion of a wide range of patients, the collection of a large amount of treatment and outcome data, and the active participation of front-line clinicians in the planning and implementation of the study. These factors all add to the ecologic validity of the PSROP.

Regarding internal validity, the authors discuss several recent investigations8, 9 comparing outcomes for randomized controlled trials (RCTs) and observational studies. These studies suggest that well-described cohort and case-control investigations produce results that are similar to RCTs. These investigations, however, examined cohort and case-control studies that involved comparison or control groups, masked recording of outcomes, and included dependent measures with established reliability and validity. As an observational study, the PSROP does not include many of these design features. The PSROP results must be carefully examined in relation to the potential limitations associated with observational investigations. This is particularly true in view of the absence of a comprehensive description of the limitations of the PSROP. In discussing the combined series of analyses included in the PSROP, the only limitation identified in the article by DeJong4 is the failure to collect follow-up data across all the participating sites.

The remainder of this commentary describes potential limitations associated with observational studies and briefly discusses how these relate to the PSROP. My comments are directed at areas of potential concern relevant to prospective cohort studies. These concerns are selection bias, observational bias, confounds, and interpretation.

Back to Article Outline

Selection bias 

Selection bias occurs when there is a preferential inclusion of subjects with certain treatment outcomes.10 In cohort studies, this usually occurs when information is less likely to be collected or analyzed from subjects who have better (or worse) outcomes. In the PSROP, the potential for selection bias is subtle because 2 cohorts (eg, treatment vs control) were not followed. In several of the analyses reported in the PSROP, however, 1 subgroup of patients is compared with another subgroup. For example, patients who received early therapy were compared with patients receiving later therapy, or patients receiving new antidepressant drugs were compared with patients administered older medications. In some cases, patients in various case-mix groups were selected for analysis and others were excluded. Patients in these subgroups may have differed in ways unknown to the investigators and not adjusted for in the statistical analyses (see Confounds below).

Back to Article Outline

Observation bias 

Observation or information bias is associated with measurement error that can be introduced in various ways.10 One strength of the PSROP is the involvement of front-line clinicians in the development of the data collection instrument and actual data gathering and recording. The participation of therapists ensures the outcomes will be clinically relevant, but also introduces a potential bias. The therapists collecting and recording data are aware of the study, its purpose, and its goals, and this may (unconsciously) affect how they treat patients and/or record data. This is frequently referred to as the Hawthorne or experimenter effect, and there is a large literature on how it can impact study outcomes.11, 12 Observation bias is known to be a potential problem in stroke rehabilitation cohort-type investigations. A previous meta-analysis13 of stroke rehabilitation studies found an interaction between design quality and study outcome that was moderated by whether or not the outcome was blindly recorded. There was no difference in outcomes between RCTs and nonrandomized studies when both sets of trials used blind recoding of the primary outcome. There were significant differences, however, when blind recording was not used, with the nonrandomized cohort studies reporting larger effect sizes and more statistically significant results.

Another area of observation bias that deserves mention relates to recording large amounts of data from medical records. No information on the accuracy or consistency of abstracting this information from the medical or other health care records is provided.

Back to Article Outline

Confounds 

Confounding occurs when 2 factors are associated with each other, and the effect of one is confused with or distorted by the effect of the other. Confounding variables differ from effect modifiers or moderator variables, but both influence outcomes.10 Confounding variables are usually controlled by manipulation of the study design or statistical methods. In the PSROP, confounding is controlled statistically. Statistical control requires that the confounding variables are known and measured. In clinical cohort studies, there are usually not enough subjects included or variables measured to statistically control for many confounding factors. Strengths of the CPI approach include the recruitment of a large and heterogeneous sample of patients and measurement of a substantial number of potential confounding variables. The Comprehensive Severity Index (CSI) was used to control for differences in patient severity of illness, including comorbidities. The CSI is described as an age- and disease-specific measure of physiologic and psychologic complexity comprised of over 2000 signs, symptoms, and physical findings.14 Little operational detail is provided about the CSI and its use in stroke rehabilitation. The cited references focus on acute care hospitalization with an emphasis on pediatrics. We do not know what variables went into the CSI, how the variables were weighted, or how missing data were handled. In the results from the summary article on early and more aggressive therapy,15 the CSI did not enter into several of the regression equations, suggesting that admission FIM and other severity-related variables shared substantial variance with CSI scores. The usefulness of the CSI relative to other potential severity adjustors such as function-related groups, and admission FIM instrument motor and/or cognitive ratings requires further investigation, particularly in view of the investigator and/or respondent burden for collecting the substantial amount of information required by the CSI.

In some analyses, general statements are made that variables were controlled for in the regression equations, but the method of control is not always clear. We do not know the ratio of variables to subjects, if interactions were tested, or if all assumptions were met for complex regression models. While the overall sample is relatively large for a clinical study, many of the regression analyses appear to include smaller numbers of subjects. More than 120 variables are listed in appendix 1, yet some of the regression models described in table 8 include fewer than 120 subjects.15

Back to Article Outline

Interpretation 

A final issue involves the level of inference that can be drawn from the data collected and analyzed in the PSROP. The article by Horn et al15 includes an excellent discussion comparing and contrasting RCTs and observational studies. I agree with the need for a broader approach to research design and the inclusion of methods other than RCTs in the generation of evidence-based knowledge for rehabilitation. While I am sympathetic to the idea that there are multiple approaches to establishing valid scientific information, we must recognize the limitations on inferences that can be drawn from a single observational study. Observational investigations, including the PSROP, provide data on associations among variables. These associations are important and can lead to improvements in practice if they are replicated and validated. Data from a single observational study do not allow the investigators to make causal inferences.10 There are many statements in the PSROP articles that imply causality. This confusion about association versus causation is reflected in the statement of the PSROP’s principle research question presented by DeJong et al: “[W]hat impact does each stroke rehabilitation activity or intervention, both individually and collectively, have on patient outcomes on discharge controlling for patient differences including medical and functional status on admission? (emphasis added).4(p3) Based on the PSROP design, this question would be better phrased as, “What is the association between each stroke rehabilitation… .” There are numerous instances where “impact,” “affect,” “influence,” and “responsible for” are used to describe the connection between variables and outcomes. There are also many examples throughout the articles where the terms association and relationship are used appropriately. Cumulatively, however, the inconsistent use of terms implying both causation and association contribute to the authors’ proposing recommendations for changes in practice that, in my opinion, are premature based on associational data from 1 study (sample).

The issues of interpretation and implications for clinical practice are directly related to establishing a research foundation for evidenced-based rehabilitation. The evidence provided by the PSROP investigation would be considered level 3 or 4 using the Center for Evidence-Based Medicine criteria.16 The importance of using appropriate levels of evidence to guide clinical practice has recently been illustrated in the radical change in practice recommendations on the use of hormone replacement therapy in postmenopausal women. Practice guidelines changed dramatically when large randomized trials did not support the findings of earlier observational studies.17

Back to Article Outline

Conclusions 

The limitations outlined above are essential to consider in evaluating the PSROP findings, but they should not detract from the importance of this research effort. The PSROP is an impressive and valuable addition to the scientific literature in stroke rehabilitation. The study provides crucial new findings and expands our understanding of the rehabilitation process applied to persons with stroke. We are all keenly aware that any investigation, particularly one as large and complex as the PSROP, will have limitations. These limitations should be recognized in order to help interpret the findings and better plan future research. The PSROP investigators have opened the lid of stroke rehabilitation’s black box. Thanks to their efforts, we have the opportunity to peer into the black box and begin the exciting challenge of exploring its contents.

Back to Article Outline

References 

  1. American Heart Association . Heart disease and stroke statistics—2005 update . 2004; Dallas: AHA
  2. Murray CJ , Lopez AD . Alternative projections of mortality and disability by cause 1990-2020 (Global Burden of Disease Study) . Lancet . 1997;349:1498–1504
  3. Rothwell P . Lack of epidemiological data on secondary stroke prevention . Lancet Neurol . 2005;4:518–519
  4. DeJong G , Horn SD , Conroy B , Nichols D , Healton EB . Opening the black box of poststroke rehabilitation (stroke rehabilitation patients, processes and outcomes) . Arch Phys Med Rehabil . 2005;86(12(Suppl 2):S1–S7
  5. Barnes M , Dobkin B , Bogousslavsky J . Recovery after stroke . New York: Cambridge Univ Pr; 2005;
  6. Bode RK , Heinemann AW , Semik P , Mallinson T . Patterns of therapy activities across length of stay and impairment levels (peering inside the “black box” of inpatient stroke rehabilitation) . Arch Phys Med Rehabil . 2004;85:1901–1908
  7. Teasell RW , Foley NC , Bhogal SK , Speechley MR . An evidence-based review of stroke rehabilitation . Top Stroke Rehabil . 2003;10:29–58
  8. Concato J , Shah N , Horwitz RI . Randomized, controlled trials, observational studies, and the hierarchy of research designs . N Engl J Med . 2000;342:1887–1892
  9. Ioannidis JP , Haidich AB , Pappa M , et al.   Comparison of evidence of treatment effects in randomized and nonrandomized studies . JAMA . 2001;286:821–830
  10. Rothman K , Greenland S . Modern epidemiology . 2nd. Philadelphia: Lippincott: Williams & Wilkins; 1998;
  11. Rosenthal R . Experimenter effects in behavioral research . New York: Irvington; 1976;
  12. Learman LA , Avorn J , Everitt DE , Rosenthal R . Pygmalion in the nursing home. The effects of caregiver expectations on patient outcomes . J Am Geriatr Soc . 1990;38:797–803
  13. Ottenbacher KJ , Jannell S . The results of clinical trials in stroke rehabilitation research . Arch Neurol . 1993;50:37–44
  14. Gassaway J , Horn SD , DeJong G , Smout RJ , Clark C , James R . Applying the clinical practice improvement approach to stroke rehabilitation (methods used and baseline results) . Arch Phys Med Rehabil . 2005;86(12(Suppl 2):S16–S33
  15. Horn SD , DeJong G , Smout RJ , Gassaway J , James R , Conroy B . Stroke rehabilitation patients, practice, and outcomes (is earlier and more aggressive therapy better?) . Arch Phys Med Rehabil . 2005;86(12(Suppl 2):S101–S114
  16. Sackett DL , Straus S , Richardson S , Rosenberg W , Haynes RB . Evidence-Based medicine (how to practice and teach EBM) . 2nd. Edinburgh: Churchill Livingstone; 2000;
  17. Col N , Pauker S . The discrepancy between observational studies and randomized trials of menopausal hormone therapy (did expectations shape experience) . Ann Intern Med . 2003;139:923–929

 No commercial party having a direct financial interest in the results of the research supporting this article has or will confer a benefit upon the author(s) or upon any organization with which the author(s) is/are associated.

PII: S0003-9993(05)01275-X

doi:10.1016/j.apmr.2005.10.001

Archives of Physical Medicine and Rehabilitation
Volume 86, Issue 12, Supplement , Pages 121-123, December 2005

Article Tools