Advertisement

Improving Measurement Methods in Rehabilitation: Core Concepts and Recommendations for Scale Development

  • Craig A. Velozo
    Correspondence
    Correspondence to Craig A. Velozo, PhD, Dept of Occupational Therapy, College of Public Health and Health Professions, University of Florida, PO Box 100164, Gainesville, FL 32610
    Affiliations
    Rehabilitation Outcomes Research Center, North Florida/South Georgia Veterans Health System, Gainesville, FL

    Department of Occupational Therapy, College of Public Health and Health Professions, University of Florida, Gainesville, FL
    Search for articles by this author
  • Ronald T. Seel
    Affiliations
    Crawford Research Institute and Brain Injury Program, Shepherd Center, Atlanta, GA
    Search for articles by this author
  • Susan Magasi
    Affiliations
    Department of Medical Social Sciences, Northwestern University Feinberg School of Medicine, Chicago, IL
    Search for articles by this author
  • Allen W. Heinemann
    Affiliations
    Department of Physical Medicine and Rehabilitation, Northwestern University Feinberg School of Medicine, Rehabilitation Institute of Chicago, Chicago, IL
    Search for articles by this author
  • Sergio Romero
    Affiliations
    Rehabilitation Outcomes Research Center, North Florida/South Georgia Veterans Health System, Gainesville, FL

    Department of Occupational Therapy, College of Public Health and Health Professions, University of Florida, Gainesville, FL
    Search for articles by this author

      Abstract

      Velozo CA, Seel RT, Magasi S, Heinemann AW, Romero S. Improving measurement methods in rehabilitation: core concepts and recommendations for scale development.
      Validated measurement scales are essential to evaluating clinical outcomes and conducting meaningful and reliable research. The purpose of this article is to present the clinician and researcher with a contemporary 8-stage framework for measurement scale development based on a mixed-methods qualitative and quantitative approach. Core concepts related to item response theory are presented. Qualitative methods are described to conceptualize scale constructs; obtain patient, family, and other stakeholder perspectives; and develop item pools. Item response theory statistical methodologies are presented, including approaches for testing the assumptions of unidimensionality, local independence, monotonicity, and indices of model fit. Lastly, challenges faced by scale developers in implementing these methodologies are discussed. While rehabilitation research has recently started to apply mixed-methods qualitative and quantitative methodologies to scale development, these approaches show considerable promise in advancing rehabilitation measurement.

      Key Words

      List of Abbreviations:

      CAT (computerized adaptive testing), CFA (confirmatory factor analysis), DIF (differential item functioning), ICF (International Classification of Functioning, Disability and Health), IRT (item response theory), logit (log odds unit), NIH (National Institutes of Health), PROMIS (Patient-Reported Outcomes Measurement Information System)
      To read this article in full you will need to make a payment

      Subscribe:

      Subscribe to Archives of Physical Medicine and Rehabilitation
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

      1. U.S. Department of Health and Human Services. Research plan for the National Center for Medical Rehabilitation Research. NIH Publication No. 93–3509 1993. Bethesda: National Institutes of Health.

        • National Institute on Disability and Rehabilitation Research (NIDRR)
        Disability and rehabilitation research projects and centers program—Rehabilitation Research and Training Centers (RRTCs)—employment policy and measurement.
        Federal Register. 2010; 75
        • Silverstein B.
        • Fisher W.P.
        • Kilgore K.M.
        • Harley J.P.
        • Harvey R.F.
        Applying psychometric criteria to functional assessment in medical rehabilitation: II.
        Arch Phys Med Rehabil. 1992; 73: 507-518
        • Linacre J.M.
        • Heinemann A.W.
        • Wright B.D.
        • Granger C.V.
        • Hamilton B.B.
        The structure and stability of the Functional Independence Measure.
        Arch Phys Med Rehabil. 1994; 75: 127-132
        • Haley S.M.
        • Ludlow L.H.
        • Coster W.J.
        Pediatric Evaluation of Disability Inventory: clinical interpretation of summary scores using Rasch rating scale methodology.
        Phys Med Rehabil Clin N Am. 1993; 4: 529-540
        • Cella D.
        • Yount S.
        • Rothrock N.
        • et al.
        The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years.
        Med Care. 2007; 45: S3-S11
        • Agency for Healthcare Research and Quality
        AHRQ Patient Centered Outcomes Research Institutional Award (K12) 2011.
        (Accessed March 4, 2012)
        • Institute P-COR
        PCORI funding announcement pilot projects grants 2011.
        (Accessed March 3, 2012)
        • U.S. Department of Health and Human Services FDA Center for Drug Evaluation and Research, U.S. Department of Health and Human Services FDA Center for Biologics Evaluation and Research and U.S. Department of Health and Human Services FDA Center for Devices and Radiological Health
        Guidance for industry: patient-reported outcome measures: use in medical product development to support labeling claims.
        (Accessed March 4, 2012.)
        • Frank G.
        • Polkinghorne D.
        Qualitative research in occupational therapy: from the first to the second generation.
        OTJR (Thorofare N J). 2010; 30: 51-57
        • Ohman A.
        Qualitative methodology for rehabilitation research.
        J Rehabil Med. 2005; 37: 273-280
        • Lasch K.E.
        • Marquis P.
        • Vigneux M.
        • et al.
        PRO development: rigorous qualitative research as the crucial foundation.
        Qual Life Res. 2010; 19: 1087-1096
        • Onwuegbuzie A.
        • Bustamante R.M.
        • Nelson J.A.
        Mixed research as a tool for developing quantitative instruments.
        J Mix Methods Res. 2010; 4: 56-77
        • DeWalt D.A.
        • Rothrock N.
        • Yount S.
        • Stone A.A.
        Evaluation of item candidates: the PROMIS qualitative item review.
        Med Care. 2007; 45: S12-S21
        • Reeve B.B.
        • Fayers P.
        Applying item response theory modeling for evaluating questionnaire item and scale properties.
        in: Fayers P. Hays R. 2nd ed. Oxford University Pr, Oxford2005: 55-73
        • Wright B.D.
        • Linacre J.M.
        Observations are always ordinal; measurements, however, must be interval.
        Arch Phys Med Rehabil. 1989; 70: 857-860
        • Wright B.D.
        • Andrich D.A.
        Rasch and Wright: the early years.
        Rasch Meas Trans. 1987; : 1-4
        • Samejima F.
        Estimation of latent ability using a response pattern of graded scores.
        Psychometrika Monogr. 1969; 17: 1-100
        • Bond T.G.
        • Fox C.M.
        Applying the Rasch model: fundamental measurement in the human sciences.
        Lawrence Erlbaum Associates, Mahwah2007
        • van der Linden W.J.
        • Hambleton R.K.
        Handbook of modern item response theory.
        Springer, New York1997
        • Hambleton R.K.
        Principles and selected applications of item response theory.
        in: Linn R.L. Educational measurement. 3rd ed. Macmillan, New York1989: 147-200
        • Andrich D.
        Controversy and the Rasch model: a characteristic of incompatible paradigms?.
        Med Care. 2004; 42: I7-I16
        • Wainer H.
        • Dorans N.
        • Eignor D.
        • et al.
        Computerized adaptive testing: a primer.
        Qual Life Res. 2001; 10: 733-734
        • Rosas S.R.
        • Camphausen L.C.
        The use of concept mapping for scale development and validation in evaluation.
        Eval Program Plann. 2007; 30: 125-135
        • Trochim M.K.
        • Trochim W.M.K.
        Concept mapping for planning and evaluation.
        Sage Publications, Thousand Oaks2007
        • Hsu C.-C.
        The Delphi technique: making sense of consensus.
        Practical Assess Res Eval. 2007; 12: 1-8
        • Strauss A.L.
        • Corbin J.M.
        Basics of qualitative research: techniques and procedures for developing grounded theory.
        2nd ed. Sage Publications, Thousand Oaks1998
        • Brod M.
        • Tesler L.E.
        • Christensen T.L.
        Qualitative research and content validity: developing best practices based on science and experience.
        Qual Life Res. 2009; 18: 1263-1278
        • Patrick D.L.
        • Burke L.B.
        • Gwaltney C.J.
        • et al.
        Content validity—establishing and reporting the evidence in newly developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO Good Research Practices Task Force report: part 2—assessing respondent understanding.
        Value Health. 2011; 14: 978-988
        • Blaikie N.W.H.
        Designing social research: the logic of anticipation.
        Polity Pr, Malden2000
        • Bowden G.
        Naturalistic inquiry and the saturation concept: a research note.
        Qual Res. 2008; 8: 137-152
        • Charmaz K.
        Constructing ground theory: a practical guide through qualitative analysis.
        Sage, Thousand Oaks2006
        • Stone M.H.
        Steps in item construction.
        Rasch Meas Trans. 1977; 11: 559
        • Velozo C.A.
        • Wang Y.
        • Lehman L.
        • Wang J.H.
        Utilizing Rasch measurement models to develop a computer adaptive self-report of walking, climbing, and running.
        Disabil Rehabil. 2008; 30: 458-467
        • DeMaio T.J.
        • Rothgeb J.M.
        Cognitive interviewing techniques in the lab and in the field.
        in: Schwarz N. Sudman S. Answering questions: methodology for determining cognitive and communicative processes in survey research. 1st ed. Jossey-Bass, San Francisco1996: 177-195
        • Willis G.B.
        Cognitive interviewing: a tool for improving questionnaire design.
        Sage Publications, Thousand Oaks2005
        • Wang W.C.
        • Chen C.T.
        Item parameter recovery, standard error estimates, and fit statistics of the Winsteps program for the family of Rasch models.
        Educ Psychol Meas. 2005; 65: 376-404
        • Reise S.P.
        • Yu J.Y.
        Parameter recovery in the graded response model using multilog.
        J Educ Meas. 1990; 27: 133-144
        • Everitt B.S.
        Multivariate analysis: the need for data and other problems.
        Br J Psychiatry. 1975; 126: 237-240
        • Reeve B.B.
        • Hays R.D.
        • Bjorner J.B.
        • et al.
        Psychometric evaluation and calibration of health-related quality of life item banks: plans for the Patient-Reported Outcomes Measurement Information System (PROMIS).
        Med Care. 2007; 45: S22-S31
        • Nunnally J.C.
        • Bernstein I.H.
        Psychometric theory.
        3rd ed. McGraw Hill, New York1994
        • Linacre J.M.
        Detecting multidimensionality: which residual data-type works best?.
        J Outcome Meas. 1998; 2: 266-283
        • Linacre J.M.
        Optimizing rating scale category effectiveness.
        J Appl Meas. 2002; 3: 85-106
        • Wright B.D.
        • Stone M.H.
        Best test design.
        MESA Pr, Chicago1979
        • Khalid M.N.
        An overview of statistical approaches for assessing model fit.
        Int J Educ Psychol Assess. 2011; 8: 69-87
        • Woodbury M.L.
        • Velozo C.A.
        • Richards L.G.
        • Duncan P.W.
        • Studenski S.
        • Lai S.M.
        Dimensionality and construct validity of the Fugl-Meyer Assessment of the upper extremity.
        Arch Phys Med Rehabil. 2007; 88: 715-723
        • Johnston M.V.
        • Shawaryn M.A.
        • Malec J.
        • Kreutzer J.
        • Hammond F.M.
        The structure of functional and community outcomes following traumatic brain injury.
        Brain Inj. 2006; 20: 391-407
        • Avery L.M.
        • Russell D.J.
        • Raina P.S.
        • Walter S.D.
        • Rosenbaum P.L.
        Rasch analysis of the Gross Motor Function Measure: validating the assumptions of the Rasch model to create an interval-level measure.
        Arch Phys Med Rehabil. 2003; 84: 697-705
        • Wright B.D.
        • Masters G.N.
        Rating scale analysis.
        Mesa Pr, Chicago1982
        • Rose M.
        • Bjorner J.B.
        • Becker J.
        • Fries J.F.
        • Ware J.E.
        Evaluation of a preliminary physical function item bank supported the expected advantages of the Patient-Reported Outcomes Measurement Information System (PROMIS).
        J Clin Epidemiol. 2008; 61: 17-33
        • Hays R.D.
        • Morales L.S.
        • Reise S.P.
        Item response theory and health outcomes measurement in the 21st century.
        Med Care. 2000; 38: II28-II42
        • Dallmeijer A.J.
        • Dekker J.
        • Roorda L.D.
        • et al.
        Differential item functioning of the Functional Independence Measure in higher performing neurological patients.
        J Rehabil Med. 2005; 37: 346-352
        • Jette A.M.
        • Haley S.M.
        • Tao W.
        • et al.
        Prospective evaluation of the AM-PAC-CAT in outpatient rehabilitation settings.
        Phys Ther. 2007; 87: 385-398
        • Hart D.L.
        • Wang Y.C.
        • Stratford P.W.
        • Mioduski J.E.
        Computerized adaptive test for patients with knee impairments produced valid and responsive measures of function.
        J Clin Epidemiol. 2008; 61: 1113-1124
        • Hart D.L.
        • Wang Y.C.
        • Stratford P.W.
        • Mioduski J.E.
        Computerized adaptive test for patients with foot or ankle impairments produced valid and responsive measures of function.
        Qual Life Res. 2008; 17: 1081-1091
        • Hart D.L.
        • Wang Y.C.
        • Cook K.F.
        • Mioduski J.E.
        A computerized adaptive test for patients with shoulder impairments produced responsive measures of function.
        Phys Ther. 2010; 90: 928-938
        • Haley S.M.
        • Ni P.
        • Ludlow L.H.
        • Fragala-Pinkham M.A.
        Measurement precision and efficiency of multidimensional computer adaptive testing of physical functioning using the Pediatric Evaluation of Disability Inventory.
        Arch Phys Med Rehabil. 2006; 87: 1223-1229
        • del Toro C.M.
        • Bislick L.P.
        • Comer M.
        • et al.
        Development of a short form of the Boston naming test for individuals with aphasia.
        J Speech Lang Hear Res. 2011; 54: 1089-1100
        • Quality Metric, Inc
        Headache Impact Test.
        (Accessed March 5, 2012)
        • Velozo C.A.
        • Woodbury M.L.
        Translating measurement findings into rehabilitation practice: an example using Fugl-Meyer Assessment-Upper Extremity with patients following stroke.
        J Rehabil Res Dev. 2011; 48: 1211-1222
        • Stout J.L.
        • Gorton III, G.E.
        • Novacheck T.F.
        • et al.
        Rasch analysis of items from two self-report measures of motor function: determination of item difficulty and relationships with children's ability levels.
        Dev Med Child Neurol. 2012; 54: 443-450
        • Divgi D.R.
        Does the Rasch model really work for multiple-choice items—not if you look closely.
        J Educ Meas. 1986; 23: 283-298
        • Fries J.F.
        • Krishnan E.
        • Rose M.
        • Lingala B.
        • Bruce B.
        Improved responsiveness and reduced sample size requirements of PROMIS physical function scales with item response theory.
        Arthritis Res Ther. 2011; 13: R147