Reducing Barriers to Patient-Reported Outcome Measures for People With Cognitive Impairments

Published:April 08, 2017DOI:


      • The design of patient-reported outcome measures (PROMs) may pose a barrier to use by patients with cognitive impairments.
      • PROMs can be designed to reduce barriers for patients with cognitive impairments.
      • We discuss PROM design features: content, layout, and administration procedures.
      • The effect of PROM design on patients with cognitive impairments requires further study.


      The field of rehabilitation has increasingly called for the use of patient-reported outcome measures (PROMs) in research and practice. Given that many rehabilitation patients present with conditions associated with cognitive impairments, it is imperative to reduce barriers to PROM use for this population. The purpose of this article is to develop a comprehensive understanding of cognitive accessibility that can prospectively inform the design of PROMs. We put forth the following definition of cognitive accessibility for PROMs: cognitive accessibility is present when assessment design anticipates respondent variability in cognitive abilities and, to the greatest extent possible, reduces cognitive demands and/or supports cognitive processes to enable respondents with a range of cognitive abilities to interpret and respond to assessment items as intended. Our operationalization of cognitive accessibility in measurement in the field of rehabilitation is informed by 2 assumptions: (1) cognitive accessibility results from an interaction between the individual's capacities and the demands of the assessment and assessment context, and (2) individuals with cognitive impairments have the right to be involved in decisions about their lives, including health care decisions. This article proposes 3 design features that can be optimized for cognitive accessibility: content, layout, and administration procedures. We end with a discussion of next steps that the field of rehabilitation measurement can undertake to advance our understanding of cognitive accessibility.


      List of abbreviations:

      PRO (patient-reported outcome), PROM (patient-reported outcome measure)
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Archives of Physical Medicine and Rehabilitation
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Kocher R.
        • Emanuel E.J.
        • DeParle N.A.
        The Affordable Care Act and the future of clinical medicine: the opportunities and challenges.
        Ann Intern Med. 2010; 153: 536-539
        • Food and Drug Administration
        Guidance for industry: patient-reported outcome measures: use in medical product development to support labeling claims.
        Fed Regist. 2009; 74: 65132-65133
        • Ahmed S.
        • Berzon R.A.
        • Revicki D.A.
        • et al.
        The use of patient-reported outcomes (PRO) within comparative effectiveness research: implications for clinical practice and health care policy.
        Med Care. 2012; 50: 1060-1070
        • Cella D.
        • Hahn E.A.
        • Jensen S.E.
        • et al.
        Patient-reported outcomes in performance measurement.
        Research Triangle Institute Press, Research Triangle2015
        • National Quality Forum
        Patient reported outcomes (PROs) in performance measurement.
        National Quality Forum, Washington (DC)2013
        • Rios D.
        • Magasi S.
        • Novak C.
        • Harniss M.
        Conducting accessible research: including people with disabilities in public health, epidemiological, and outcomes studies.
        Am J Public Health. 2016; 106: 2137-2144
        • Velozo C.A.
        • Seel R.T.
        • Magasi S.
        • Heinemann A.W.
        • Romero S.
        Improving measurement methods in rehabilitation: core concepts and recommendations for scale development.
        Arch Phys Med Rehabil. 2012; 93: S154-S163
        • Butler M.
        • Kane R.L.
        • Larson S.
        • Jeffery M.M.
        • Grove M.
        Quality improvement measurement of outcomes for people with disabilities. Closing the quality gap: revisiting the state of the science: evidence report/technology assessment executive summary no. 208.
        Agency for Healthcare Research and Quality, Rockville2012 (AHRQ Pub. No. 12(13)-E013-1)
        • Andresen E.M.
        Criteria for assessing the tools of disability outcomes research.
        Arch Phys Med Rehabil. 2000; 81: S15-S20
        • American Psychological Association
        Guidelines for assessment of and intervention with persons with disabilities.
        Am Psychol. 2012; 67: 43-62
        • American Educational Research Association
        • American Psychological Association
        • National Council on Measurement in Education
        Standards for educational and psychological testing.
        American Educational Research Association, Washington (DC)1999
        • Meyers A.R.
        • Andresen E.M.
        Enabling our instruments: accommodation, universal design, and access to participation in research.
        Arch Phys Med Rehabil. 2000; 81: S5-S9
        • Harniss M.
        • Amtmann D.
        • Cook D.
        • Johnson K.
        Considerations for developing interfaces for collecting patient-reported outcomes that allow the inclusion of individuals with disabilities.
        Med Care. 2007; 45: S48-S54
        • Dowler R.N.
        • Harrington D.L.
        • Haaland K.Y.
        • Swanda R.M.
        • Fee F.
        • Fiedler K.
        Profiles of cognitive functioning in chronic spinal cord injury and the role of moderating variables.
        J Int Neuropsychol Soc. 1997; 3: 464-472
        • Gorelick P.B.
        • Scuteri A.
        • Black S.E.
        • et al.
        Vascular contributions to cognitive impairment and dementia: a statement for healthcare professionals from the American Heart Association/American Stroke Association.
        Stroke. 2011; 42: 2672-2713
        • Cheng G.
        • Huang C.
        • Deng H.
        • Wang H.
        Diabetes as a risk factor for dementia and mild cognitive impairment: a meta-analysis of longitudinal studies.
        Intern Med J. 2012; 42: 484-491
        • World Health Organization
        International classification of functioning, disability and health: ICF.
        World Health Organization, Geneva2001
        • Dolan R.P.
        • Burling K.S.
        • Rose D.
        • et al.
        Universal Design for Computer-Based Testing (UD-CBT) guidelines.
        Pearson, Boulder, CO2010
        • Beddow P.A.
        Accessibility theory for enhancing the validity of test results for students with special needs.
        Intl J Disabil Dev Educ. 2012; 59: 97-111
      1. Beddow PA, Elliott SN, Kettler RJ. Test accessibility: item reviews and lessons learned from four state assessments. Education Research International 2013;952704. Available at: Accessed May 30, 2017

        • Beddow P.A.
        TAMI accessibility rating matrix: technical manual.
        Vanderbilt University, Nashville2010
        • Tourangeau R.
        Cognitive sciences and survey methods.
        in: Jabine T.B. Straf M.L. Tanur J.M. Tourangeau R. Cognitive aspects of survey methodology: Building a bridge between disciplines. National Academy Press, Washington, DC1984: 73-100
        • Fujiura G.
        • RRTC Expert Panel on Health Measurement
        Self-reported health of people with intellectual disability.
        Intellect Dev Disabil. 2012; 50: 352-369
        • Iwarsson S.
        • Ståhl A.
        Accessibility, usability and universal design—positioning and definition of concepts describing person-environment relationships.
        Disabil Rehabil. 2003; 25: 57-66
        • Kettler R.J.
        • Dickenson T.S.
        • Bennett H.L.
        • et al.
        Enhancing the accessibility of high school science tests: a multistate experiment.
        Except Child. 2012; 79: 91-106
        • Yalon-Chamovitz S.
        Invisible access needs of people with intellectual disabilities: a conceptual model of practice.
        Intellect Dev Disabil. 2009; 47: 395-400
        • Kettler R.J.
        • Rodriguez M.C.
        • Bolt D.M.
        • Elliott S.N.
        • Beddow P.A.
        • Kurz A.
        Modified multiple-choice items for alternate assessments: reliability, difficulty, and differential boost.
        Applied Measurement in Education. 2011; 24: 210-234
        • Hartley S.L.
        • MacLean W.E.
        A review of the reliability and validity of Likert-type scales for people with intellectual disability.
        J Intellect Disabil Res. 2006; 50: 813-827
        • Cuskelly M.
        • Moni K.
        • Lloyd J.
        • Jobling A.
        Reliability of a method for establishing the capacity of individuals with an intellectual disability to respond to Likert scales.
        J Intellect Dev Disabil. 2013; 38: 318-324
      2. Nielsen-Bohlman L. Panzer A.M. Kindig D.A. Committee on Health Literacy Health literacy: a prescription to end confusion. National Academies Press, Washington (DC)2004
        • Rose D.
        • Meyer A.
        • Hitchcock C.
        The universally designed classroom: accessible curriculum and digital technologies.
        Harvard University Press, Cambridge2005
        • Abedi J.
        • Leon S.
        • Kao J.
        • et al.
        Accessible reading assessments for students with disabilities: the role of cognitive, grammatical, lexical, and textual/visual features.
        National Center for Research on Evaluation, Standards, and Student Testing (CRESST), Los Angeles, CA2011 (CRESST Report 785)
        • National Institutes of Health
        • Northwestern University
        NIH Toolbox: reasonable accommodation guidelines.
        National Institutes of Health, Bethesda, MA2012
        • Clark R.C.
        • Nguyen F.
        • Sweller J.
        Efficiency in learning: evidence-based guidelines to manage cognitive load.
        John Wiley & Sons, San Francisco2011
        • Finlay W.M.
        • Lyons E.
        Methodological issues in interviewing and using self-report questionnaires with people with mental retardation.
        Psychol Assess. 2001; 13: 319
        • Sturgess J.
        • Rodger S.
        • Ozanne A.
        A review of the use of self-report assessment with young children.
        Br J Occup Ther. 2002; 65: 108-116
        • Eddy L.
        • Khastou L.
        • Cook K.F.
        • Amtmann D.
        Item selection in self-report measures for children and adolescents with disabilities: lessons from cognitive interviews.
        J Pediatr Nurs. 2011; 26: 559-565
        • Rapley M.
        The social construction of intellectual disability.
        Cambridge University Press, Cambridge2004
        • Finlay W.M.
        • Lyons E.
        Acquiescence in interviews with people who have mental retardation.
        Ment Retard. 2002; 40: 14-29
        • Nind M.
        • Seale J.
        Concepts of access for people with learning difficulties: towards a shared understanding.
        Disabil Soc. 2009; 24: 273-287
        • Schwarz N.
        Self-reports: how the questions shape the answers.
        Am Psychol. 1999; 54: 93-105
        • Sprangers M.A.
        • Schwartz C.E.
        Integrating response shift into health-related quality of life research: a theoretical model.
        Soc Sci Med. 1999; 48: 1507-1515
        • Schwartz C.E.
        Applications of response shift theory and methods to participation measurement: a brief history of a young field.
        Arch Phys Med Rehabil. 2010; 91: S38-S43
        • Rizzo M.R.
        • Marfella R.
        • Barbieri M.
        • et al.
        Relationships between daily acute glucose fluctuations and cognitive performance among aged type 2 diabetic patients.
        Diabetes Care. 2010; 33: 2169-2174
        • Witjas T.
        • Kaphan E.
        • Azulay J.
        • et al.
        Nonmotor fluctuations in Parkinson's disease: frequent and disabling.
        Neurology. 2002; 59: 408-413
        • Brega A.G.
        • Barnard J.
        • Mabachi N.M.
        • et al.
        AHRQ health literacy universal precautions toolkit, second edition. (Prepared by Colorado Health Outcomes Program, University of Colorado Anschutz Medical Campus under Contract No. HHSA290200710008, TO#10).
        Agency for Healthcare Research and Quality, Rockville2015 (AHRQ Publication No. 15-0023-EF)
        • Mencap
        Am I making myself clear?: Mencap's guidelines for accessible writing.
        Mencap, London2000
        • Wehmeyer M.L.
        Self-determination and individuals with severe disabilities: re-examining meanings and misinterpretations.
        Res Pract Persons Severe Disabl. 2005; 30: 113-120
        • Goodley D.
        'Learning difficulties', the social model of disability and impairment: challenging epistemologies.
        Disabil Soc. 2001; 16: 207-231
      3. Noll S. Trent J.W. Mental retardation in America: a historical reader. New York University Press, New York2004
        • Magasi S.
        Infusing disability studies into the rehabilitation sciences.
        Top Stroke Rehabil. 2008; 15: 283-287
        • Longmore P.K.
        Why I burned my book and other essays on disability.
        Temple University Press, Philadelphia2003
        • Centers for Disease Control and Prevention
        Simply put: a guide for creating easy-to-understand materials.
        Centers for Disease Control and Prevention, Atlanta2009
        • Friedman M.G.
        • Bryen D.N.
        Web accessibility design recommendations for people with cognitive disabilities.
        Technol Disabil. 2007; 19: 205
        • White-Koning M.
        • Arnaud C.
        • Bourdet-Loubère S.
        • Bazex H.
        • Colver A.
        • Grandjean H.
        Subjective quality of life in children with intellectual impairment–how can it be assessed?.
        Dev Med Child Neurol. 2005; 47: 281-285
      4. Diagram Center. General guidelines. Available at: Retrieved August 1, 2014.

      5. The Inclusive Learning Design Handbook. Accessible standardized testing: functional considerations. Available at: Retrieved August 1, 2014.

      6. Rudd RE. Guidelines for creating materials: resources for developing and assessing materials. Harvard University. Available at: Accessed May 2014.

        • Kettler R.J.
        Testing accommodations: theory and research to inform practice.
        Int J Disabil Dev Educ. 2012; 59: 53-66
        • Kramer J.
        • Walker R.
        • Cohn E.S.
        • et al.
        Striving for shared understandings: therapists' perspectives of the benefits and dilemmas of using a child self-assessment.
        OTJR (Thorofare N J). 2012; 32: S48-S58
        • Leach M.J.
        Rapport: a key to treatment success.
        Complement Ther Clin Pract. 2005; 11: 262-265
        • Street R.L.
        • Makoul G.
        • Arora N.K.
        • Epstein R.M.
        How does communication heal? Pathways linking clinician–patient communication to health outcomes.
        Patient Educ Couns. 2009; 74: 295-301
        • Kramer J.M.
        • Liljenquist K.
        • Coster W.J.
        Validity, reliability, and usability of the Pediatric Evaluation of Disability Inventory-Computer Adaptive Test for autism spectrum disorders.
        Dev Med Child Neurol. 2016; 58: 255-261
        • DeWalt D.A.
        • Rothrock N.
        • Yount S.
        • Stone A.A.
        Evaluation of item candidates: the PROMIS qualitative item review.
        Med Care. 2007; 45: S12
        • Collins D.
        Pretesting survey instruments: an overview of cognitive methods.
        Qual Life Res. 2003; 12: 229-238
        • Kramer J.M.
        • Schwartz A.
        Refining the Pediatric Evaluation of Disability Inventory–Patient-Reported Outcome (PEDI-PRO) item candidates: interpretation of a self-reported outcome measure of functional performance by young people with neurodevelopmental disabilities.
        Dev Med Child Neurol. 2017 Jun 5; ([E-pub ahead of print])
        • Jette A.M.
        • Haley S.M.
        Contemporary measurement techniques for rehabilitation outcomes assessment.
        J Rehabil Med. 2005; 37: 339-345
        • Betts L.
        • Hartley J.
        The effects of changes in the order of verbal labels and numerical values on children's scores on attitude and rating scales.
        Br Educ Res J. 2012; 38: 319-331
        • Christian L.M.
        • Parsons N.L.
        • Dillman D.A.
        Designing scalar questions for web surveys.
        Sociological Methods & Research. 2009; 37: 393-425