Advertisement

Towards a Better Measure of Brain Injury Outcome: New Measures or a New Metric?

  • Jacob Kean
    Affiliations
    Center for Health Information and Communication, Department of Veterans Affairs, Veterans Health Administration, Health Services Research and Development Service CIN 13-416, Richard L. Roudebush VA Medical Center, Indianapolis, IN

    Department of Physical Medicine and Rehabilitation, Indiana University School of Medicine, Indianapolis, IN

    Regenstrief Institute, Indianapolis, IN
    Search for articles by this author
  • James F. Malec
    Correspondence
    Corresponding author James F. Malec, PhD, ABPP-Cn, Rp, Professor and Research Director, Department of Physical Medicine and Rehabilitation, Indiana University School of Medicine and Rehabilitation Hospital of Indiana, 4141 Shore Dr, Indianapolis, IN 46254.
    Affiliations
    Department of Physical Medicine and Rehabilitation, Indiana University School of Medicine, Indianapolis, IN

    Rehabilitation Hospital of Indiana, Indianapolis, IN
    Search for articles by this author
Published:April 11, 2014DOI:https://doi.org/10.1016/j.apmr.2014.03.023

      List of abbreviations:

      BI (brain injury), CDE (Common Data Elements), DRS (Disability Rating Scale), GOS (Glasgow Outcome Scale), GOSE (Glasgow Outcome Scale–Extended), SES (socioeconomic status), TBIMS (Traumatic Brain Injury Model Systems)
      An audio podcast accompanies this article.
      The Glasgow Outcome Scale (GOS), its successor the Glasgow Outcome Scale-Extended (GOSE),
      • Wilson J.L.
      • Pettigrew L.E.
      • Teasdale G.M.
      Structured interviews for the Glasgow Outcome Scale and the extended Glasgow Outcome Scale: guidelines for their use.
      and the Disability Rating Scale (DRS)
      • Rappaport M.
      • Hall K.
      • Hopkins K.
      • Belleza T.
      • Cope D.
      Disability Rating Scale for severe head trauma: coma to community.
      represent a desire for a single outcome score that can be used to track the progression of a patient with brain injury (BI) from coma to community. Although the GOS/GOSE and DRS have been rightly criticized for their psychometric and other limitations,
      • Hall K.
      • Cope D.
      • Rappaport M.
      Glasgow Outcome Scale and Disability Rating Scale: comparative usefulness in following recovery in traumatic head injury.
      • Hall K.M.
      • Bushnik T.
      • Lakisic-Kazazic B.
      • Wright J.
      • Cantagallo A.
      Assessing traumatic brain injury outcome measures for long-term follow-up of community-based individuals.
      • Hall K.M.
      • Hamilton B.B.
      • Gordon W.A.
      • Zasler N.D.
      Characteristics and comparisons of functional assessment indices: Disability Rating Scale, Functional Independence Measure, and Functional Assessment Measure.
      these scales span the major domains of the International Classification of Functioning, Disability and Health—that is, bodily functions, activities, and participation.
      World Health Organization
      International Classification of Functioning, Disability and Health: ICF.
      These early outcomes scales also map directly onto the conceptual domains identified as representing outcome in 2010 by the Common Data Elements (CDE) Task Force (fig 1). Two of the areas targeted by the CDE Task Force but not explicitly mentioned by the GOSE and DRS—behavioral function and neuropsychological impairment—may be considered more detailed and objective measurements of cognitive limitations that are clearly targeted as outcome indicators by both the GOSE and DRS. The additional domains identified by the CDE Task Force (health economic measures, patient-reported outcomes) represent, in the former case, an outcome indicator that does not reside within the person with BI, and in the latter case, a domain that overlaps with other domains.
      Figure thumbnail gr1
      Fig 1Brain injury outcome domains and recommended measures. Abbreviations: CHART-SF, The Craig Handicap Assessment and Reporting Technique–Short Form; PART, Participation Assessment with Recombined Tools; TBI, traumatic brain injury.
      The Traumatic Brain Injury Model Systems (TBIMS) study group has implicitly endorsed this conceptualization of outcome by retaining the GOS/GOSE and DRS throughout its 30-year history. The TBIMS group adopted the Satisfaction With Life Scale
      • Diener E.
      • Emmons R.A.
      • Larsen R.J.
      • Griffin S.
      The Satisfaction With Life Scale.
      to represent important content that was absent from the GOSE or DRS. Although included in both the GOSE and DRS, specific indicators of independent living and productive activity that appear basic to our idea of outcome are also recorded separately in the TBIMS longitudinal database. In short, the domains described in figure 1 appear to represent a conceptualization of outcome that has been informed by early outcome scales and has withstood the test of time, with ongoing and recent endorsement by the TBIMS study group and by the CDE Task Force.

      Reflective outcome measurement

      The GOSE and DRS are reflective measures. Reflective measures are composites of individual items (ie, observed variables) that are indicators believed to describe or reflect an underlying construct (ie, latent variable).
      • Bollen K.
      • Lennox R.
      Conventional wisdom on measurement: a structural equation perspective.
      In contrast, formative measures, as we explain in more detail below, are composed of indicators believed to determine or form the construct. Socioeconomic status (SES) is an example of a formative measure. In reflective measurement, latent constructs are typically measured as the aggregation of reflective indicators (ie, items in a scale) because the indirectness of the indicators requires that several aspects be considered together to reasonably approximate a construct. The ability of a reflective measure to differentiate ability is related to the number of items and their interrelatedness. Because good measures reliably differentiate performance and better measures do so with greater precision, the recipe for creating the best measures is to develop a large number of interrelated items that reflect the full range of ability on the construct of interest.
      However, improving reflective composites of global outcome such as GOSE and DRS using this approach is difficult because outcome is so broad and likely multidimensional. In other words, completely capturing a broad conceptual domain along a continuum “from coma to community” can come at the cost of compromised measurement precision. Satisfaction with life, for example, which is perhaps as important as or more important to outcome than activities and participation, is a domain found by Corrigan and Bogner
      • Corrigan J.D.
      • Bogner J.
      Latent factors in measures of rehabilitation outcomes after traumatic brain injury.
      as well as in our own unpublished work to be statistically orthogonal (ie, uncorrelated) to other outcome domains. Other outcome domains depicted in figure 1 correlate to varying degrees with one another. The complex relationships among these various outcome domains pose a significant challenge to the establishment of a single outcome scale to track recovery.
      One approach to improving on available but coarse reflective measures is the use of advanced multidimensional and hierarchical psychometric models to account for the complex relationships among latent traits. One might, for example, collate extant items from available reflective measures, such as those listed in figure 1. We are currently collaborating on a TBIMS module project of this type. A challenge for efforts such as this is that the psychometric qualities of these available measures vary considerably. Fitting demanding new models to older scales not designed to be consistent with such frameworks can result in poor fit and violations of model assumptions.
      • Reise S.P.
      • Horan W.P.
      • Blanchard J.J.
      The challenges of fitting an item response theory model to the Social Anhedonia Scale.
      Developing new multidimensional reflective measures from the ground up may yield more precise measures but is time and resource intensive, requiring domain mapping, qualitative item review or development, item library construction, cognitive interviewing, psychometric evaluation and calibration using advanced models, and large sample sizes (eg, 2000+).
      • Cella D.
      • Yount S.
      • Rothrock N.
      • et al.
      The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years.
      These advanced reflective models may more realistically approximate the latent construct of interest. However, the resulting measures can be difficult to interpret since they consist of multiple distinct but correlated unidimensional subscales.

      Formative outcome measurement

      Another option is to create a formative, rather than reflective, composite. Whereas a reflective measure is composed of observed indicators believed to reflect the manifestation of an underlying latent construct, a formative measure is composed of observable indicators believed to determine the construct.
      • Bollen K.
      • Lennox R.
      Conventional wisdom on measurement: a structural equation perspective.
      In a formative model, the indicators (ie, items) do not reflect the latent construct because it makes more conceptual sense to view the construct as the reflection of the items (rather than vice versa as in reflective measures). Bollen and Lennox
      • Bollen K.
      • Lennox R.
      Conventional wisdom on measurement: a structural equation perspective.
      offer the example of SES as a construct that is realized as the aggregation of 4 formative indicators: education, occupational prestige, income, and neighborhood. In this case, the direction of determination flows from the 4 indicators to the concept of SES rather than the reverse, as would be the case for a reflective measure. A formative measure may combine various types of indicators (eg, categorical, ordinal, continuous) that may or may not be correlated. For example, the components of SES are not highly correlated, but each informs the assessment of an individual's SES. That is, a college professor may be of relatively high SES on the basis of education and occupational prestige despite relatively low income, whereas a lottery winner may achieve greater SES based on income alone. A formative measure, in simplest form, is based on a regression model in which the dependent variable is the construct of interest and the explanatory (predictive) variables are observable indicators.
      • Bollen K.
      • Lennox R.
      Conventional wisdom on measurement: a structural equation perspective.
      For this reason, including variables in the model that are correlated with the latent trait, in this case outcome, but not with each other, should improve the model.
      Table 1 shows criteria offered by Jarvis et al
      • Jarvis C.B.
      • Mackenzie S.B.
      • Podsakoff P.M.
      A critical review of construct indicators and measurement model misspecification in marketing and consumer research.
      to help researchers consider whether the relationship between measures and the construct of interest is reflective or formative. The first set of questions relates to the direction of determination between the construct and its indicators, the second to the interchangeability of the indicators, the third to the covariance among indicators, and the fourth to the nomothethic net, or antecedents and consequences, of the indicators. Despite these criteria, the general advice is that most constructs can be measured reflectively and that researchers should opt for reflective measurement, if possible.
      • Howell R.D.
      • Breivik E.
      • Wilcox J.B.
      Reconsidering formative measurement.
      Nevertheless, for constructs such as SES, the consensus seems to be that formative measurement is a better option than a reflective measure with indicators such as “How high are you up the social ladder?” (as suggested in Borsboom et al
      • Borsboom D.
      • Mellenbergh G.J.
      • van Heerden J.
      The concept of validity.
      [p1069]).
      Table 1Decision rules for determining whether a construct is formative or reflective
      Decision RulesFormative ModelReflective Model
      • 1.
        Direction of causality from construct to measure implied by the conceptual definition
        • Are the indicators (items) (a) defining characteristics or
        • (b) manifestations of the construct?
        • Would changes in the indicators/items cause changes in the construct or not?
        • Would changes in the construct cause changes in the indicators?
      Direction of causality is from items to construct.

      Indicators are defining characteristics of the construct.

      Changes in the indicators should cause changes in the construct.

      Changes in the construct do not cause changes in the indicators.
      Direction of causality is from construct to items.

      Indicators are manifestations of the construct.

      Changes in the indicator should not cause changes in the construct.

      Changes in the construct do cause changes in the indicators.
      • 2.
        Interchangeability of the indicators/items
        • Should the indicators have the same or similar content?
        • Do the indicators share a common theme?
        • Would dropping one of the indicators alter the conceptual domain of the construct?
      Indicators need not be interchangeable.

      Indicators need not have the same or similar content.

      Indicators need not share a common theme.

      Dropping an indicator may alter the conceptual domain of the construct.
      Indicators should be interchangeable.

      Indicators should have the same or similar content.

      Indicators need not share a common theme.

      Dropping an indicator should not alter the conceptual domain of the construct.
      • 3.
        Covariation among the indicators
      • Should a change in one of the indicators be associated with changes in the other indicators?
      Not necessary for indicators to covary with each other

      Not necessarily
      Indicators are expected to covary with each other.

      Yes
      • 4.
        Nomological net of the construct indicators
        • Are the indicators/items expected to have the same antecedents and consequences?
      Nomological net for the indicators may differ.

      Indicators are not required to have the same antecedents and consequences.
      Nomological net for the indicators should not differ.

      Indicators are required to have the same antecedents and consequences.
      NOTE. From Jarvis et al
      • Jarvis C.B.
      • Mackenzie S.B.
      • Podsakoff P.M.
      A critical review of construct indicators and measurement model misspecification in marketing and consumer research.
      (2003). Reprinted with permission.
      Despite the potential advantages, many have been critical of formative measurement.
      • Howell R.D.
      • Breivik E.
      • Wilcox J.B.
      Reconsidering formative measurement.
      One issue is the challenge of determining how to aggregate and weight components, which has been shown to have pronounced effects on the ordering of cases of interest.
      • Jacobs R.
      • Goddard M.
      • Smith P.C.
      How robust are hospital ranks based on composite performance measures?.
      A related issue is the need to include reflective measures in the measurement model as determinants of the formative construct to achieve identification. In other words, without also modeling reflective measures “caused” directly or indirectly by the formative construct, there are an infinite number of solutions to a formative measurement model. However, the choice of reflective measures can affect the loadings of the formative elements on the composite, which arguably affects the meaning of the construct itself.
      • Edwards J.R.
      The fallacy of formative measurement.
      Unlike reflective measures, too great of a relationship among indicators (ie, multicollinearity) poses a computational problem for the statistical model. In addition, there is no psychometric theory to guide validity testing of formative measures. Finally, some consider it inappropriate to test theory using formative measures because formative measurement models assume that the theoretical construct is created by rather than reflected in its measures.
      Nonetheless, a formative measure may make conceptual sense for BI outcome, as it does for SES. The statistical orthogonality between satisfaction with life and activities and participation after BI indicates no relationship between these outcome domains. This dissociation is difficult to handle in a reflective model, which expects indicators to covary. A formative outcome measure, in contrast, would account for the independent influence of life satisfaction as well as activity and functioning on outcome without assuming covariation among these outcome indicators. A formative measure allows “mixing apples and oranges” (ie, uncorrelated indicators) in situations where the measurement goal is analogous to recording servings of fruit consumed each day. With its diversity of composite indicators, outcome may be such a mix of apples and oranges.
      More concretely, a formative measure might be constructed using available measures and indicators. Assuming that underlying neuropsychological, psychological and behavioral impairment would be captured by indicators of activity limitations, a measure that combines the GOSE, DRS, FIM subscales, Satisfaction With Life Scale, and employment and independent living status may sufficiently cover the outcome domains identified historically by the field (see fig 1). Our research group is currently engaged in examining the practicality and viability of such a measure. As mentioned previously, the greatest challenges in developing such a measure may be achieving model identification and verifying its external validity.
      Although challenges to the development of a formative outcome measurement for BI are significant and the topic of formative measurement is a controversial one, developing precision reflective outcome measures de novo is no less challenging and certainly more resource intensive. A formative outcome measure for BI is achievable in a shorter period since it is based on available data and has the practical advantages of a single score and accommodation of complex relationships among measures. Treating outcome as a construct determined by indicators may also make more conceptual sense than modeling outcome as a latent construct. For these and other reasons, formative measurement may deserve consideration in physical medicine and rehabilitation clinical and research communities.

      Supplementary data

      References

        • Wilson J.L.
        • Pettigrew L.E.
        • Teasdale G.M.
        Structured interviews for the Glasgow Outcome Scale and the extended Glasgow Outcome Scale: guidelines for their use.
        J Neurotrauma. 1998; 15: 573-585
        • Rappaport M.
        • Hall K.
        • Hopkins K.
        • Belleza T.
        • Cope D.
        Disability Rating Scale for severe head trauma: coma to community.
        Arch Phys Med Rehabil. 1982; 63: 118-123
        • Hall K.
        • Cope D.
        • Rappaport M.
        Glasgow Outcome Scale and Disability Rating Scale: comparative usefulness in following recovery in traumatic head injury.
        Arch Phys Med Rehabil. 1985; 66: 35
        • Hall K.M.
        • Bushnik T.
        • Lakisic-Kazazic B.
        • Wright J.
        • Cantagallo A.
        Assessing traumatic brain injury outcome measures for long-term follow-up of community-based individuals.
        Arch Phys Med Rehabil. 2001; 82: 367-374
        • Hall K.M.
        • Hamilton B.B.
        • Gordon W.A.
        • Zasler N.D.
        Characteristics and comparisons of functional assessment indices: Disability Rating Scale, Functional Independence Measure, and Functional Assessment Measure.
        J Head Trauma Rehabil. 1993; 8: 60-74
        • World Health Organization
        International Classification of Functioning, Disability and Health: ICF.
        World Health Organization, Geneva2001
        • Diener E.
        • Emmons R.A.
        • Larsen R.J.
        • Griffin S.
        The Satisfaction With Life Scale.
        J Pers Assess. 1985; 49: 71-75
        • Bollen K.
        • Lennox R.
        Conventional wisdom on measurement: a structural equation perspective.
        Psychol Bull. 1991; 110: 305-314
        • Corrigan J.D.
        • Bogner J.
        Latent factors in measures of rehabilitation outcomes after traumatic brain injury.
        J Head Trauma Rehabil. 2004; 19: 445-458
        • Reise S.P.
        • Horan W.P.
        • Blanchard J.J.
        The challenges of fitting an item response theory model to the Social Anhedonia Scale.
        J Pers Assess. 2011; 93: 213-224
        • Cella D.
        • Yount S.
        • Rothrock N.
        • et al.
        The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years.
        Med Care. 2007; 45: S3-11
        • Jarvis C.B.
        • Mackenzie S.B.
        • Podsakoff P.M.
        A critical review of construct indicators and measurement model misspecification in marketing and consumer research.
        J Consum Res. 2003; 30: 199-218
        • Howell R.D.
        • Breivik E.
        • Wilcox J.B.
        Reconsidering formative measurement.
        Psychol Methods. 2007; 12: 205-218
        • Borsboom D.
        • Mellenbergh G.J.
        • van Heerden J.
        The concept of validity.
        Psychol Rev. 2004; 111: 1061-1071
        • Jacobs R.
        • Goddard M.
        • Smith P.C.
        How robust are hospital ranks based on composite performance measures?.
        Med Care. 2005; 43: 1177-1184
        • Edwards J.R.
        The fallacy of formative measurement.
        Organ Res Methods. 2011; 14: 370-388