Abstract
Objective
To present challenges in assessment of applied cognition and the results of differential
item functioning (DIF) analyses used to inform the development of a computerized adaptive
test (CAT).
Design
Measurement evaluation cohort study. DIF analyses of 107 items were conducted across
educational, age, and sex groups. DIF hypotheses informed the evaluation of the results.
Setting
Hospital-based rehabilitation from a single hospital system.
Participants
A total of 2216 hospitalized patients (N=2216).
Interventions
Not applicable.
Main Outcome Measures
Applied cognition item pool from multiple sources.
Results
Many items were hypothesized to show DIF, particularly for age. Information was moderately
high in the lower (cognitive disability) tail of the distribution, but some items
were not informative. Reliability estimates were high (>0.89) across all studied groups,
regardless of estimation method. There were 35 items with DIF of high magnitude and
19 with accompanying supportive hypotheses.
Conclusions
A key clinical tool in inpatient rehabilitation medicine is assessment of applied
functional cognitive ability to inform patient-centered rehabilitation strategies
to improve function. This was the first study to evaluate measurement equivalence
of the applied cognition item pool across large samples of hospitalized patients.
Although about one-third of the item pool evidenced DIF or low discrimination, results
supported placement of most items into the bank and its use across groups differing
in education, age, and sex. Six items were classified with salient DIF, defined as
consistent DIF of high magnitude and or impact, with confirmatory directional DIF
hypotheses, generated by content experts. These were recommended for adjustment or
removal from the bank; 4 were deleted from the bank and 2 had lowered CAT exposure
(administration frequency) rates. Many items hypothesized to show DIF contained content
measuring constructs other than applied cognition such as physical frailty, perceptual
difficulties, or skills reflective of greater educational attainment. Challenges in
measurement of this construct are discussed.
Keywords
List of abbreviations:
CAT (computerized adaptive test), DIF (differential item functioning), FAMCAT (Functional Assessment in Acute Care Multidimensional Computerized Adaptive Test), FKGL (Flesch-Kincaid Grade Level), FREI (Flesch-Kincaid Reading Ease Index), IRT (item response theory)To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Archives of Physical Medicine and RehabilitationAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- Subjective cognitive decline among adults aged ≥ 45 years–United States, 2015-2016.MMWR Morb Mortal Wkly Rep. 2018; 67: 753-757
- Self-reported increased confusion or memory loss and associated functional difficulties among adults aged ≥ 60 years–21 states, 2011.MMWR Morb Mortal Wkly Rep. 2013; 62: 345-350
- Test bias in a cognitive test: differential item functioning in the CASI.Stat Med. 2004; 23: 241-256
- Differential item functioning in the Short Portable Mental Status Questionnaire.Res Aging. 2005; 27: 355-384
- Performance of cognitive tests among different racial/ethnic and education groups: findings of differential item functioning and possible item bias.J Ment Health Aging. 2001; 7: 79-89
- Use of the Mini-Mental State Examination (MMSE) in a community population of mixed ethnicity.J Nerv Ment Dis. 1986; 174: 607-614
- Education and sex differences in the Mini-Mental State Examination: effects of differential item functioning.J Gerontol. 2002; 57B: 548-558
- Item bias in cognitive screening measures: comparisons of elderly White, Afro-American, Hispanic and high and low education subgroups.J Clin Epidemiol. 1995; 48: 473-483
- Final Report to the National Institute of Mental Health, the Hispanic Alzheimer’s Research Project (HARP).San Diego State University, San Diego1991
- Multiple, correlated covariates associated with differential item functioning (DIF): accounting for language DIF when education levels differ across languages.Ageing Res. 2011; 2: 19-25
- Measurement invariance of neuropsychological tests in diverse older persons.Neuropsychology. 2011; 25: 260-269
- Modern psychometric methods for detection of differential item functioning: application to cognitive assessment measures.Stat Med. 2000; 19: 1651-1683
- Measurement equivalence of the Patient Reported Outcomes Measurement Information System (PROMIS) applied cognition–general concerns, short forms in ethnically diverse groups.Psychol Test Assess Model. 2016; 58: 255-307
- The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH roadmap cooperative group during its first two years.Med Care. 2007; 45: S3-S11
- Psychometric evaluation and calibration of Health-Related Quality of Life Items Banks: plans for the Patient-Reported Outcome Measurement Information System (PROMIS).Med Care. 2007; 45: S22-S31
- The influence of diversity on the measurement of functional impairment: an international validation of the Amsterdam IADL Questionnaire in eight countries.Alzheimers Dement (Amst). 2020; 12e12021
Cheville AL, Wang C, Weiss D, et al. Design and methods to develop and validate a multidimensional computerized adaptive test (MCAT) to direct delivery of hospital-based rehabilitation: the Functional Assessment in Acute Care MCAT (FAMCAT). Arch Phys Med Rehabil.
- Activity outcome measurement for post-acute care.Med Care. 2004; 42: I-49-I61
- A new readability yardstick.J Appl Psychol. 1948; 32: 221-223
- Estimation of latent ability using a response pattern of graded scores.Psychometrika Monogr Suppl. 1969; 34: 100-114
- IRTPRO: flexible, multidimensional, multiple categorical IRT modeling.Scientific Software International Inc, Chicago2011
- A re-examination of Lord’s Wald test for differential item functioning using item response theory and modern error estimation [dissertation].Chapel Hill: University of North Carolina at Chapel Hill Library, 2008
- Applications of item response theory to practical testing problems.Lawrence Erlbaum, Hillsdale, NJ1980
- Applications of item response theory to the examination of the psychometric properties and differential item functioning of the CARE Dementia Diagnostic Scale among samples of Latino, African-American and White non-Latino elderly.Res Aging. 2000; 22: 738-773
- The Langer-improved Wald test for DIF testing with multiple groups: evaluation and comparison to two-group IRT.Educ Psychol Meas. 2013; 73: 532-547
- lordif: an R package for detecting differential item functioning using iterative hybrid ordinal logistic regression/item response theory and Monte Carlo simulations.J Stat Softw. 2011; 39: 1-30
- Test theory: a unified treatment.L. Erlbaum Associates, Mahwah, NJ1999
- Coefficient alpha and the internal structure of tests.Psychometrika. 1951; 16: 297-334
- Ordinal versions of coefficient alpha and theta for Likert rating scales.J Mod Appl Stat Methods. 2007; 6: 21-29
- The Parkinson Disease Cognitive Functional Rating Scale (PD-CFRS): a brief and specific instrument to rate the impact of PD cognitive symptoms on daily function.Neurology. 2012; 78: 6-054
- Assessing cognitive impairment using PROMIS(®) Applied Cognition-Abilities Scales in a medical outpatient sample.Psychiatry Res. 2015; 226: 169-172
- The measurement of everyday cognition: development and validation of a short form of the Everyday Cognition Scales.Alzheimers Dement. 2011; 7: 593-601
- Differential item functioning analysis with ordinal logistic regression techniques: DIFdetect and difwithpar.Med Care. 2006; 44: S115-S123
- Differential item functioning on the Mini-Mental State Examination: an application of the Mantel-Haenszel and standardization procedures.Med Care. 2006; 44: S107-S114
- Identification of differential item functioning using item response theory and the likelihood-based model comparison approach: applications to the Mini-Mental State Examination.Med Care. 2006; 44: S134-S142
- Identification of measurement differences between English and Spanish language versions of the Mini-Mental State Examination: detecting differential item functioning using MIMIC modeling.Med Care. 2006; 44: S124-S133
- Item and scale differential functioning of the Mini-Mental State Exam assessed using the differential item and test functioning (DFIT) framework.Med Care. 2006; 44: S143-S151
- Differential item functioning in the Mini-Mental State Examination in English- and Spanish-speaking older adults.Psychol Aging. 1997; 12: 718-725
- Differential item functioning in the Danish translation of the SF-36.J Clin Epidemiol. 1998; 51: 1189-1202
- Cross-cultural adaptation of the Systemic Lupus Erythematosus Quality of Life Questionnaire into Chinese.Arthritis Rheum. 2007; 57: 980-985
- Measuring cognition in a geriatric outpatient clinic: Rasch analysis of the Montreal Cognitive Assessment.J Geriatr Psychiatry Neurol. 2009; 22: 151-160
- French-English cross-linguistic comparison and diagnostic impact of the AD-8 dementia screening questionnaire in a geriatric assessment clinic.Dement Geriatr Cogn Disord. 2010; 29: 265-274
- Differential item functioning by survey language among older Hispanics enrolled in Medicare managed care: a new method for anchor item selection.Med Care. 2011; 49: 461-468
- Differential item functioning of the English and Chinese versions of the Fagerstrom Test for Nicotine Dependence.Addict Behav. 2009; 34: 125-133
- A comparison of item response theory-based methods for examining differential item functioning in object naming test by language of assessment among older Latinos.Psychol Test Assess Model. 2011; 53: 440-460
- Age and education correction of Mini-Mental State Examination for English and Spanish-speaking elderly.Neurology. 1996; 46: 700-706
- Screening scales for dementia: toward reconciliation of conflicting cross-cultural findings.Int J Geriatr Psychiatry. 1992; 7: 105-113
- Cognitive assessment among minority elderly: possible test bias.J Ment Health Aging. 2001; 7: 91-118
- Differential item functioning (DIF) and the Mini-Mental Status Examination (MMSE): overview, sample, and issues of translation.Med Care. 2006; 44: S95-S106
- Characteristics of a two-stage screen for incident dementia.J Clin Epidemiol. 2003; 56: 1038-1045
- Effects of adjustment on the case-finding potential of cognitive tests.J Clin Epidemiol. 1996; 49: 691-695
- The association between educational attainment and mental status examinations: of etiological significance for senile dementias or not?.J Chronic Dis. 1986; 39: 171-175
- Methodological issues in screening for dementia: the problem of education adjustment.J Chronic Dis. 1986; 39: 163-170
- Diagnosis of dementia in a heterogeneous population: development of a neuropsychological paradigm-based diagnosis of dementia and quantified correction for the effects of education.Arch Neurol. 1992; 49: 453-460
- Cognitive correlates with functional outcomes after anoxic brain injury: a case-controlled comparison with traumatic brain injury.Brain Inj. 2011; 25: 35-43
- Neuropsychological assessment and the Disability Rating Scale (DRS): a concurrent validity study.Brain Inj. 2000; 14: 719-724
- The predictive validity of a brief inpatient neuropsychologic battery for persons with traumatic brain injury.Arch Phys Med Rehabil. 2008; 89: 950-957
- Measures of cognitive functioning in the 1994-2000 Second Longitudinal Study of Aging.National Center for Health Statistics, Hyattsville, MD2008
- The association between cognitive impairment and functional outcome in hospitalized older patients: a systematic review and meta-analysis.Age Ageing. 2017; 46: 559-567
- Association of functional, cognitive, and psychological measures with 1-year mortality in patients undergoing major surgery.JAMA Surg. 2020; 155: 412-418
Article info
Publication history
Published online: February 05, 2021
Footnotes
Supported in part by a Eunice Kennedy Shriver National Institutes of Child Health and Human Development of the National Institutes of Health award (no. R01HD079439) to the Mayo Clinic in Rochester Minnesota through subcontracts to the University of Minnesota and the University of Washington, the National Institute of Child Health and Human Development (grant no. 1R01HD079439-01A1), the National Institute on Aging Mount Sinai Pepper Center (grant no. 1P30AG028741), and the Columbia University Alzheimer’s Disease Resource Center for Minority Aging Research (grant no. 1P30AG059303). The content is solely the responsibility of the authors and does not necessarily reflect the official views of the National Institutes of Health.
Disclosures: none.
This paper is part of a supplement from the American Congress of Rehabilitation Medicine.
Identification
Copyright
2022 Published by Elsevier Inc. on behalf of the American Society of Echocardiography.
