Performance of an Item Response Theory-Based Computer Adaptive Test in Identifying Functional Decline

Published:February 27, 2012DOI:


      Cheville AL, Yost KJ, Larson DR, Dos Santos K, O'Byrne MM, Chang MT, Therneau TM, Diehn FE, Yang P. Performance of an item response theory-based computer adaptive test in identifying functional decline.


      To achieve a low respondent burden and increase the responsiveness of functional measurement by using an item response theory-based computer adaptive test (CAT), the Activity Measure for Post-Acute Care (AM-PAC) CAT.


      Two-year prospective cohort study.


      Telephonic assessments from a quaternary medical center.


      Patients (N=311) with late-stage lung cancer (LC).


      Monthly assessments for up to 2 years. Disease progression was determined via record abstraction. Anchor-based responsiveness techniques were used to compare AM-PAC-CAT score changes between global rating of change (GRC) question response levels, as well as between intervals when adverse clinical events or symptom worsening did and did not occur. Distribution-based responsiveness assessments included calculation of the standardized effect size (SES) and standardized response mean (SRM).

      Main Outcome Measures

      AM-PAC-CAT, symptom numerical rating scales, and a GRC.


      Administration time averaged 112 seconds over 2543 interviews. AM-PAC-CAT score changes became more positive as GRC responses reflected more improved states: a lot worse (−11.62), a little worse (−1.92), the same (−.10), a little better (1.01), and a lot better (2.82). Score changes were negative when associated with adverse clinical events. The SES and SRM for score differences between 1 to 2 and 9 to 10 months prior to death were −.87 and −1.13, respectively. The minimally important difference estimate was defined by the mean CAT session SE at 2.0.


      The AM-PAC-CAT imposes a low, <2-minute, respondent burden, and distribution- and anchor-based methods suggest that is moderately responsive in patients with late-stage LC.

      Key Words

      List of Abbreviations:

      AM-PAC (Activity Measure for Post-Acute Care), AM-PAC-CAT (Activity measure for post acute care computer adaptive test), AUC (area under receiver operating characteristic curve), CAT (computer adaptive test), EMR (electronic medical record), GEE (generalized estimating equation), GRC (global rating of change), IRT (item response theory), LC (lung cancer), MID (minimally important difference), PRO (patient-reported outcome), PROMIS (Patient-Reported Outcomes Measurement Information System), RR (responsiveness retrospective), SES (standardized effect size), SRM (standardized response mean)
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Archives of Physical Medicine and Rehabilitation
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Cheville A.L.
        • Beck L.A.
        • Petersen T.L.
        • Marks R.S.
        • Gamble G.L.
        The detection and treatment of cancer-related functional problems in an outpatient setting.
        Support Care Cancer. 2009; 17: 61-67
        • Cheville A.L.
        • Troxel A.B.
        • Basford J.R.
        • Kornblith A.B.
        Prevalence and treatment patterns of physical impairments in patients with metastatic breast cancer.
        J Clin Oncol. 2008; 26: 2621-2629
        • Cheville A.L.
        • Basford J.R.
        • Troxel A.B.
        • Kornblith A.B.
        Performance of common clinician- and self-report measures in assessing the function of community-dwelling people with metastatic breast cancer.
        Arch Phys Med Rehabil. 2009; 90: 2116-2124
        • Guyatt G.H.
        • Deyo R.A.
        • Charlson M.
        • Levine M.N.
        • Mitchell A.
        Responsiveness and validity in health status measurement: a clarification.
        J Clin Epidemiol. 1989; 42: 403-408
        • Eurich D.T.
        • Johnson J.A.
        • Reid K.J.
        • Spertus J.A.
        Assessing responsiveness of generic and specific health related quality of life measures in heart failure.
        Health Qual Life Outcomes. 2006; 4: 89
        • Cella D.
        • Riley W.
        • Stone A.
        • et al.
        The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005-2008.
        J Clin Epidemiol. 2010; 63: 1179-1194
        • Jette A.M.
        • Haley S.M.
        • Tao W.
        • et al.
        Prospective evaluation of the AM-PAC-CAT in outpatient rehabilitation settings.
        Phys Ther. 2007; 87: 385-398
        • Li Y.
        • Sheu C.C.
        • Ye Y.
        • et al.
        Genetic variants and risk of lung cancer in never smokers: a genome-wide association study.
        Lancet Oncol. 2010; 11: 321-330
        • Coster W.J.
        • Haley S.M.
        • Andres P.L.
        • Ludlow L.H.
        • Bond T.L.
        • Ni P.S.
        Refining the conceptual basis for rehabilitation outcome measurement: personal care and instrumental activities domain.
        Med Care. 2004; 42: I62-I72
        • Haley S.M.
        • Coster W.J.
        • Andres P.L.
        • et al.
        Activity outcome measurement for postacute care.
        Med Care. 2004; 42: I49-I61
        • Siebens H.
        • Andres P.L.
        • Pengsheng N.
        • Coster W.J.
        • Haley S.M.
        Measuring physical function in patients with complex medical and postsurgical conditions: a computer adaptive approach.
        Am J Phys Med Rehabil. 2005; 84: 741-748
        • Haley S.M.
        • Fragala-Pinkham M.
        • Ni P.
        Sensitivity of a computer adaptive assessment for measuring functional mobility changes in children enrolled in a community fitness programme.
        Clin Rehabil. 2006; 20: 616-622
        • Fitzpatrick R.
        • Ziebland S.
        • Jenkinson C.
        • Mowat A.
        • Mowat A.
        Transition questions to assess outcomes in rheumatoid arthritis.
        Br J Rheumatol. 1993; 32: 807-811
        • Cleeland C.S.
        • Mendoza T.R.
        • Wang X.S.
        • et al.
        Assessing symptom distress in cancer patients: the M.D. Anderson Symptom Inventory.
        Cancer. 2000; 89: 1634-1646
        • Buchanan D.R.
        • O'Mara A.M.
        • Kelaghan J.W.
        • Minasian L.M.
        Quality-of-life assessment in the symptom management trials of the National Cancer Institute-supported Community Clinical Oncology Program.
        J Clin Oncol. 2005; 23: 591-598
        • Sugimura H.
        • Nichols F.C.
        • Yang P.
        • et al.
        Survival after recurrent nonsmall-cell lung cancer after complete pulmonary resection.
        Ann Thorac Surg. 2007; 83 (discussion 417-8): 409-417
        • Visbal A.L.
        • Williams B.A.
        • Nichols 3rd, F.C.
        • et al.
        Gender differences in non-small-cell lung cancer survival: an analysis of 4,618 patients diagnosed between 1997 and 2002.
        Ann Thorac Surg. 2004; 78 (discussion 215): 209-215
        • Turner D.
        • Schunemann H.J.
        • Griffith L.E.
        • et al.
        Using the entire cohort in the receiver operating characteristic analysis maximizes precision of the minimal important difference.
        J Clin Epidemiol. 2009; 62: 374-379
        • Deyo R.A.
        • Diehr P.
        • Patrick D.L.
        Reproducibility and responsiveness of health status measures.
        Control Clin Trials. 1991; : 142S-158S
        • Deyo R.A.
        • Centor R.M.
        Assessing the responsiveness of functional scales to clinical change: an analogy to diagnostic test performance.
        J Chronic Dis. 1986; 39: 897-906
        • Husted J.A.
        • Cook R.J.
        • Farewell V.T.
        • Gladman D.D.
        Methods for assessing responsiveness: a critical review and recommendations.
        J Clin Epidemiol. 2000; 53: 459-468
        • Tuley M.R.
        • Mulrow C.D.
        • McMahan C.A.
        Estimating and testing an index of responsiveness and the relationship of the index to power.
        J Clin Epidemiol. 1991; 44: 417-421
        • Williamson G.M.
        • Schulz R.
        Activity restriction mediates the association between pain and depressed affect: a study of younger and older adult cancer patients.
        Psychol Aging. 1995; 10: 369-378
        • Given B.
        • Given C.
        • Azzouz F.
        • Stommel M.
        Physical functioning of elderly cancer patients prior to diagnosis and following initial treatment.
        Nurs Res. 2001; 50: 222-232
        • Meenan R.F.
        • Anderson J.J.
        • Kazis L.E.
        • et al.
        Outcome assessment in clinical trials.
        Arthritis Rheum. 1984; 27: 1344-1352
        • Given C.W.
        • Given B.A.
        • Stommel M.
        The impact of age, treatment, and symptoms on the physical and mental health of cancer patients.
        Cancer. 1994; 74: 2128-2138
        • Tao W.
        • Haley S.M.
        • Coster W.J.
        • Ni P.
        • Jette A.M.
        An exploratory analysis of functional staging using an item response theory approach.
        Arch Phys Med Rehabil. 2008; 89: 1046-1053
        • Beaton D.E.
        • Hogg-Johnson S.
        • Bombardier C.
        Evaluating changes in health status: reliability and responsiveness of five generic health status measures in workers with musculoskeletal disorders.
        J Clin Epidemiol. 1997; 50: 79-93
        • Fitzpatrick R.
        • Ziebland S.
        • Jenkinson C.
        • Mowat A.
        • Mowat A.
        A comparison of the sensitivity to change of several health status instruments in rheumatoid arthritis.
        J Rheumatol. 1993; 20: 429-436
        • Liang M.H.
        • Fossel A.H.
        • Larson M.G.
        Comparisons of five health status instruments for orthopedic evaluation.
        Med Care. 1990; 28: 632-642
        • Guyatt G.
        • Walter S.
        • Norman G.
        Measuring change over time: assessing the usefulness of evaluative instruments.
        J Chronic Dis. 1987; 40: 171-178
        • Cohen J.
        Statistical power analysis for the behavioral sciences.
        Lawrence Erlbaum Associates, Hillsdale1988
        • Revicki D.
        • Hays R.D.
        • Cella D.
        • Sloan J.
        Recommended methods for determining responsiveness and minimally important differences for patient-reported outcomes.
        J Clin Epidemiol. 2008; 61: 102-109
        • Yost K.J.
        • Eton D.T.
        • Garcia S.F.
        • Cella D.
        Minimally important differences were estimated for six Patient-Reported Outcomes Measurement Information System-Cancer scales in advanced-stage cancer patients.
        J Clin Epidemiol. 2011; 64: 507-516
        • Lunney J.R.
        • Lynn J.
        • Foley D.J.
        • Lipson S.
        • Guralnik J.M.
        Patterns of functional decline at the end of life.
        JAMA. 2003; 289: 2387-2392
        • Costantini M.
        • Beccaro M.
        • Higginson I.J.
        Cancer trajectories at the end of life: is there an effect of age and gender?.
        BMC Cancer. 2008; 8: 127
        • Liang M.H.
        Longitudinal construct validity: establishment of clinical meaning in patient evaluative instruments.
        Med Care. 2000; 38: II84-II90
        • Norman G.R.
        • Stratford P.
        • Regehr G.
        Methodological problems in the retrospective computation of responsiveness to change: the lesson of Cronbach.
        J Clin Epidemiol. 1997; 50: 869-879
        • Guyatt G.H.
        • Norman G.R.
        • Juniper E.F.
        • Griffith L.E.
        A critical look at transition ratings.
        J Clin Epidemiol. 2002; 55: 900-908
        • Cella D.
        • Hahn E.A.
        • Dineen K.
        Meaningful change in cancer-specific quality of life scores: differences between improvement and worsening.
        Qual Life Res. 2002; 11: 207-221
        • Wallace D.
        • Duncan P.W.
        • Lai S.M.
        Comparison of the responsiveness of the Barthel Index and the motor component of the Functional Independence Measure in stroke: the impact of using different methods for measuring responsiveness.
        J Clin Epidemiol. 2002; 55: 922-928
        • Latham N.K.
        • Mehta V.
        • Nguyen A.M.
        • et al.
        Performance-based or self-report measures of physical function: which should be used in clinical trials of hip fracture patients?.
        Arch Phys Med Rehabil. 2008; 89: 2146-2155
        • De Groot I.J.
        • Post M.W.
        • Van Heuveln T.
        • Van Den Berg L.H.
        • Lindeman E.
        Measurement of decline of functioning in persons with amyotrophic lateral sclerosis: responsiveness and possible applications of the Functional Independence Measure, Barthel Index, Rehabilitation Activities Profile and Frenchay Activities Index.
        Amyotroph Lateral Scler. 2006; 7: 167-172
        • Frihagen F.
        • Grotle M.
        • Madsen J.E.
        • Wyller T.B.
        • Mowinckel P.
        • Nordsletten L.
        Outcome after femoral neck fractures: a comparison of Harris Hip Score, Eq-5d and Barthel Index.
        Injury. 2008; 39: 1147-1156
        • Deyo R.A.
        • Inui T.S.
        Toward clinical applications of health status measures: sensitivity of scales to clinically important changes.
        Health Serv Res. 1984; 19: 275-289
        • Krebs E.E.
        • Bair M.J.
        • Damush T.M.
        • Tu W.
        • Wu J.
        • Kroenke K.
        Comparative responsiveness of pain outcome measures among primary care patients with musculoskeletal pain.
        Med Care. 2010; 48: 1007-1014
        • Amjadi S.S.
        • Maranian P.M.
        • Paulus H.E.
        • et al.
        Validating and assessing the sensitivity of the Health Assessment Questionnaire-Disability Index-derived Short Form-6D in patients with early aggressive rheumatoid arthritis.
        J Rheumatol. 2009; 36: 1150-1157
        • Chang E.
        • Abrahamowicz M.
        • Ferland D.
        • Fortin P.R.
        Comparison of the responsiveness of lupus disease activity measures to changes in systemic lupus erythematosus activity relevant to patients and physicians.
        J Clin Epidemiol. 2002; 55: 488-497
        • King M.T.
        A point of minimal important difference (MID): a critique of terminology and methods.
        Expert Rev Pharmacoecon Outcomes Res. 2011; 11: 171-184