Advertisement

Challenges in Measuring Applied Cognition: Measurement Properties and Equivalence of the Functional Assessment in Acute Care Multidimensional Computerized Adaptive Test (FAMCAT) Applied Cognition Item Bank

Published:February 05, 2021DOI:https://doi.org/10.1016/j.apmr.2020.12.029

      Abstract

      Objective

      To present challenges in assessment of applied cognition and the results of differential item functioning (DIF) analyses used to inform the development of a computerized adaptive test (CAT).

      Design

      Measurement evaluation cohort study. DIF analyses of 107 items were conducted across educational, age, and sex groups. DIF hypotheses informed the evaluation of the results.

      Setting

      Hospital-based rehabilitation from a single hospital system.

      Participants

      A total of 2216 hospitalized patients (N=2216).

      Interventions

      Not applicable.

      Main Outcome Measures

      Applied cognition item pool from multiple sources.

      Results

      Many items were hypothesized to show DIF, particularly for age. Information was moderately high in the lower (cognitive disability) tail of the distribution, but some items were not informative. Reliability estimates were high (>0.89) across all studied groups, regardless of estimation method. There were 35 items with DIF of high magnitude and 19 with accompanying supportive hypotheses.

      Conclusions

      A key clinical tool in inpatient rehabilitation medicine is assessment of applied functional cognitive ability to inform patient-centered rehabilitation strategies to improve function. This was the first study to evaluate measurement equivalence of the applied cognition item pool across large samples of hospitalized patients. Although about one-third of the item pool evidenced DIF or low discrimination, results supported placement of most items into the bank and its use across groups differing in education, age, and sex. Six items were classified with salient DIF, defined as consistent DIF of high magnitude and or impact, with confirmatory directional DIF hypotheses, generated by content experts. These were recommended for adjustment or removal from the bank; 4 were deleted from the bank and 2 had lowered CAT exposure (administration frequency) rates. Many items hypothesized to show DIF contained content measuring constructs other than applied cognition such as physical frailty, perceptual difficulties, or skills reflective of greater educational attainment. Challenges in measurement of this construct are discussed.

      Keywords

      List of abbreviations:

      CAT (computerized adaptive test), DIF (differential item functioning), FAMCAT (Functional Assessment in Acute Care Multidimensional Computerized Adaptive Test), FKGL (Flesch-Kincaid Grade Level), FREI (Flesch-Kincaid Reading Ease Index), IRT (item response theory)
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Archives of Physical Medicine and Rehabilitation
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Taylor C.A.
        • Bouldin E.D.
        • McGuire L.C.
        Subjective cognitive decline among adults aged ≥ 45 years–United States, 2015-2016.
        MMWR Morb Mortal Wkly Rep. 2018; 67: 753-757
        • Centers for Disease Control and Prevention
        Self-reported increased confusion or memory loss and associated functional difficulties among adults aged ≥ 60 years–21 states, 2011.
        MMWR Morb Mortal Wkly Rep. 2013; 62: 345-350
        • Crane P.K.
        • van Belle G.
        • Larson E.B.
        Test bias in a cognitive test: differential item functioning in the CASI.
        Stat Med. 2004; 23: 241-256
        • Stump T.E.
        • Monahan P.
        • McHorney C.A.
        Differential item functioning in the Short Portable Mental Status Questionnaire.
        Res Aging. 2005; 27: 355-384
        • Teresi J.A.
        • Holmes D.
        • Ramirez M.
        • Gurland B.J.
        • Lantigua R.
        Performance of cognitive tests among different racial/ethnic and education groups: findings of differential item functioning and possible item bias.
        J Ment Health Aging. 2001; 7: 79-89
        • Escobar J.L.
        • Burnam A.
        • Karno M.
        • Forsythe A.
        • Landsverk J.
        • Golding J.M.
        Use of the Mini-Mental State Examination (MMSE) in a community population of mixed ethnicity.
        J Nerv Ment Dis. 1986; 174: 607-614
        • Jones R.N.
        • Gallo J.J.
        Education and sex differences in the Mini-Mental State Examination: effects of differential item functioning.
        J Gerontol. 2002; 57B: 548-558
        • Teresi J.A.
        • Golden R.
        • Cross P.
        • Gurland B.
        • Kleinman M.
        • Wilder D.
        Item bias in cognitive screening measures: comparisons of elderly White, Afro-American, Hispanic and high and low education subgroups.
        J Clin Epidemiol. 1995; 48: 473-483
        • Valle R.
        • Hough R.
        • Kolody B.
        • Cook-Gait H.
        • Velazquez G.F.
        • Jimenez R.
        • The validation of the Blessed Mental Status Test and the Mini-Mental State Examination with a Hispanic population
        Final Report to the National Institute of Mental Health, the Hispanic Alzheimer’s Research Project (HARP).
        San Diego State University, San Diego1991
        • Gibbons L.E.
        • Crane P.K.
        • Mehta K.M.
        • et al.
        Multiple, correlated covariates associated with differential item functioning (DIF): accounting for language DIF when education levels differ across languages.
        Ageing Res. 2011; 2: 19-25
        • Mungas D.
        • Widaman K.F.
        • Reed B.R.
        • Tomaszewski Farias S.
        Measurement invariance of neuropsychological tests in diverse older persons.
        Neuropsychology. 2011; 25: 260-269
        • Teresi J.A.
        • Kleinman M.
        • Ocepek-Welikson K.
        Modern psychometric methods for detection of differential item functioning: application to cognitive assessment measures.
        Stat Med. 2000; 19: 1651-1683
        • Fieo R.
        • Ocepek-Welikson K.
        • Kleinman M.
        • et al.
        Measurement equivalence of the Patient Reported Outcomes Measurement Information System (PROMIS) applied cognition–general concerns, short forms in ethnically diverse groups.
        Psychol Test Assess Model. 2016; 58: 255-307
        • Cella D.
        • Yount S.
        • Rothrock N.
        • et al.
        The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH roadmap cooperative group during its first two years.
        Med Care. 2007; 45: S3-S11
        • Reeve B.B.
        • Hays R.D.
        • Bjorner J.B.
        • et al.
        Psychometric evaluation and calibration of Health-Related Quality of Life Items Banks: plans for the Patient-Reported Outcome Measurement Information System (PROMIS).
        Med Care. 2007; 45: S22-S31
        • Dubbelman M.A.
        • Verrijp M.
        • Facal D.
        • et al.
        The influence of diversity on the measurement of functional impairment: an international validation of the Amsterdam IADL Questionnaire in eight countries.
        Alzheimers Dement (Amst). 2020; 12e12021
      1. Cheville AL, Wang C, Weiss D, et al. Design and methods to develop and validate a multidimensional computerized adaptive test (MCAT) to direct delivery of hospital-based rehabilitation: the Functional Assessment in Acute Care MCAT (FAMCAT). Arch Phys Med Rehabil.

        • Haley S.M.
        • Coster W.J.
        • Andres P.L.
        • et al.
        Activity outcome measurement for post-acute care.
        Med Care. 2004; 42: I-49-I61
        • Flesch R.
        A new readability yardstick.
        J Appl Psychol. 1948; 32: 221-223
        • Samejima F.
        Estimation of latent ability using a response pattern of graded scores.
        Psychometrika Monogr Suppl. 1969; 34: 100-114
        • Cai L.
        • Thissen D.
        • du Toit S.H.C.
        IRTPRO: flexible, multidimensional, multiple categorical IRT modeling.
        Scientific Software International Inc, Chicago2011
        • Langer M.M.
        A re-examination of Lord’s Wald test for differential item functioning using item response theory and modern error estimation [dissertation].
        Chapel Hill: University of North Carolina at Chapel Hill Library, 2008
        • Lord F.M.
        Applications of item response theory to practical testing problems.
        Lawrence Erlbaum, Hillsdale, NJ1980
        • Teresi J.A.
        • Kleinman M.
        • Ocepek-Welikson K.
        • et al.
        Applications of item response theory to the examination of the psychometric properties and differential item functioning of the CARE Dementia Diagnostic Scale among samples of Latino, African-American and White non-Latino elderly.
        Res Aging. 2000; 22: 738-773
        • Woods C.M.
        • Cai L.
        • Wang M.
        The Langer-improved Wald test for DIF testing with multiple groups: evaluation and comparison to two-group IRT.
        Educ Psychol Meas. 2013; 73: 532-547
        • Choi S.W.
        • Gibbons L.E.
        • Crane P.K.
        lordif: an R package for detecting differential item functioning using iterative hybrid ordinal logistic regression/item response theory and Monte Carlo simulations.
        J Stat Softw. 2011; 39: 1-30
        • McDonald R.P.
        Test theory: a unified treatment.
        L. Erlbaum Associates, Mahwah, NJ1999
        • Cronbach L.J.
        Coefficient alpha and the internal structure of tests.
        Psychometrika. 1951; 16: 297-334
        • Zumbo B.D.
        • Gadermann A.M.
        • Zeisser C.
        Ordinal versions of coefficient alpha and theta for Likert rating scales.
        J Mod Appl Stat Methods. 2007; 6: 21-29
        • Kulisevsky J.
        • Fernández de Bobadilla R.
        • Pagonabarraga J.
        • et al.
        The Parkinson Disease Cognitive Functional Rating Scale (PD-CFRS): a brief and specific instrument to rate the impact of PD cognitive symptoms on daily function.
        Neurology. 2012; 78: 6-054
        • Saffer B.Y.
        • Lanting S.C.
        • Koehle M.S.
        • Klonsky E.D.
        • Iverson G.L.
        Assessing cognitive impairment using PROMIS(®) Applied Cognition-Abilities Scales in a medical outpatient sample.
        Psychiatry Res. 2015; 226: 169-172
        • Tomaszewski Farias S.
        • Mungas D.
        • Harvey D.J.
        • Simmons A.
        • Reed B.R.
        • DeCarli C.
        The measurement of everyday cognition: development and validation of a short form of the Everyday Cognition Scales.
        Alzheimers Dement. 2011; 7: 593-601
        • Crane P.K.
        • Gibbons L.E.
        • Jolley L.
        • van Belle G.
        Differential item functioning analysis with ordinal logistic regression techniques: DIFdetect and difwithpar.
        Med Care. 2006; 44: S115-S123
        • Dorans N.J.
        • Kulick E.
        Differential item functioning on the Mini-Mental State Examination: an application of the Mantel-Haenszel and standardization procedures.
        Med Care. 2006; 44: S107-S114
        • Edelen M.O.
        • Thissen D.
        • Teresi J.A.
        • Kleinman M.
        • Ocepek-Welikson K.
        Identification of differential item functioning using item response theory and the likelihood-based model comparison approach: applications to the Mini-Mental State Examination.
        Med Care. 2006; 44: S134-S142
        • Jones R.N.
        Identification of measurement differences between English and Spanish language versions of the Mini-Mental State Examination: detecting differential item functioning using MIMIC modeling.
        Med Care. 2006; 44: S124-S133
        • Morales L.S.
        • Flowers C.
        • Gutierrez P.
        • Kleinman M.
        • Teresi J.A.
        Item and scale differential functioning of the Mini-Mental State Exam assessed using the differential item and test functioning (DFIT) framework.
        Med Care. 2006; 44: S143-S151
        • Marshall S.C.
        • Mungas D.
        • Weldon M.
        • Reed B.
        • Haan M.
        Differential item functioning in the Mini-Mental State Examination in English- and Spanish-speaking older adults.
        Psychol Aging. 1997; 12: 718-725
        • Bjorner J.B.
        • Kreiner S.
        • Ware J.E.
        • Damsgaard M.T.
        • Bech P.
        Differential item functioning in the Danish translation of the SF-36.
        J Clin Epidemiol. 1998; 51: 1189-1202
        • Kong K.O.
        • Ho H.J.
        • Howe H.S.
        • Thong B.Y.
        Cross-cultural adaptation of the Systemic Lupus Erythematosus Quality of Life Questionnaire into Chinese.
        Arthritis Rheum. 2007; 57: 980-985
        • Koski L.
        • Xie H.
        • Finch L.
        Measuring cognition in a geriatric outpatient clinic: Rasch analysis of the Montreal Cognitive Assessment.
        J Geriatr Psychiatry Neurol. 2009; 22: 151-160
        • Koski L.
        • Xie H.
        • Konsztowicz S.
        • Tetteh R.
        French-English cross-linguistic comparison and diagnostic impact of the AD-8 dementia screening questionnaire in a geriatric assessment clinic.
        Dement Geriatr Cogn Disord. 2010; 29: 265-274
        • Setodji C.M.
        • Reise S.P.
        • Morales L.S.
        • Fongwa M.N.
        • Hays R.D.
        Differential item functioning by survey language among older Hispanics enrolled in Medicare managed care: a new method for anchor item selection.
        Med Care. 2011; 49: 461-468
        • Yamada H.
        • Acton G.S.
        • Tsoh J.Y.
        Differential item functioning of the English and Chinese versions of the Fagerstrom Test for Nicotine Dependence.
        Addict Behav. 2009; 34: 125-133
        • Yang F.M.
        • Heslin K.C.
        • Mehta K.M.
        • et al.
        A comparison of item response theory-based methods for examining differential item functioning in object naming test by language of assessment among older Latinos.
        Psychol Test Assess Model. 2011; 53: 440-460
        • Mungas D.
        • Marshall S.C.
        • Weldon M.
        • Haan M.
        • Reed B.R.
        Age and education correction of Mini-Mental State Examination for English and Spanish-speaking elderly.
        Neurology. 1996; 46: 700-706
        • Gurland B.
        • Wilder D.
        • Cross P.
        • Teresi J.
        • Barrett V.
        Screening scales for dementia: toward reconciliation of conflicting cross-cultural findings.
        Int J Geriatr Psychiatry. 1992; 7: 105-113
        • Ramírez M.
        • Teresi J.A.
        • Silver S.
        • Holmes D.
        • Gurland B.J.
        • Lantigua R.
        Cognitive assessment among minority elderly: possible test bias.
        J Ment Health Aging. 2001; 7: 91-118
        • Ramírez M.
        • Teresi J.
        • Holmes D.
        • Gurland B.
        • Lantigua R.
        Differential item functioning (DIF) and the Mini-Mental Status Examination (MMSE): overview, sample, and issues of translation.
        Med Care. 2006; 44: S95-S106
        • Hayden K.M.
        • Khachaturian A.S.
        • Tschanz J.T.
        • Corcoran C.
        • Nortond M.
        • Breitner J.C.S.
        • for the Cache County Study Group
        Characteristics of a two-stage screen for incident dementia.
        J Clin Epidemiol. 2003; 56: 1038-1045
        • Lindeboom J.
        • Launer L.J.
        • Schmand B.A.
        • Hooyer C.
        • Jonker C.
        Effects of adjustment on the case-finding potential of cognitive tests.
        J Clin Epidemiol. 1996; 49: 691-695
        • Berkman L.F.
        The association between educational attainment and mental status examinations: of etiological significance for senile dementias or not?.
        J Chronic Dis. 1986; 39: 171-175
        • Kittner S.J.
        • White L.R.
        • Farmer M.E.
        • Wolz M.
        Methodological issues in screening for dementia: the problem of education adjustment.
        J Chronic Dis. 1986; 39: 163-170
        • Stern Y.
        • Andrews H.
        • Pittman J.
        • et al.
        Diagnosis of dementia in a heterogeneous population: development of a neuropsychological paradigm-based diagnosis of dementia and quantified correction for the effects of education.
        Arch Neurol. 1992; 49: 453-460
        • Cullen N.K.
        • Weisz K.
        Cognitive correlates with functional outcomes after anoxic brain injury: a case-controlled comparison with traumatic brain injury.
        Brain Inj. 2011; 25: 35-43
        • Neese L.E.
        • Caroselli J.S.
        • Klaas P.
        • High Jr., W.M.
        • Becker L.J.
        • Scheibel R.S.
        Neuropsychological assessment and the Disability Rating Scale (DRS): a concurrent validity study.
        Brain Inj. 2000; 14: 719-724
        • Hanks R.A.
        • Millis S.R.
        • Ricker J.H.
        • et al.
        The predictive validity of a brief inpatient neuropsychologic battery for persons with traumatic brain injury.
        Arch Phys Med Rehabil. 2008; 89: 950-957
        • Pratt L.A.
        • Weeks J.D.
        • Goulding M.R.
        Measures of cognitive functioning in the 1994-2000 Second Longitudinal Study of Aging.
        National Center for Health Statistics, Hyattsville, MD2008
        • Hartley P.
        • Gibbins N.
        • Saunders A.
        • et al.
        The association between cognitive impairment and functional outcome in hospitalized older patients: a systematic review and meta-analysis.
        Age Ageing. 2017; 46: 559-567
        • Tang V.L.
        • Jing B.
        • Boscardin J.
        • et al.
        Association of functional, cognitive, and psychological measures with 1-year mortality in patients undergoing major surgery.
        JAMA Surg. 2020; 155: 412-418