ORIGINAL RESEARCH| Volume 102, ISSUE 11, P2125-2133, November 2021

Download started.


Indicators of Quality Rehabilitation Services for Individuals with Limited English Proficiency: A 3-Round Delphi Study


      • This study examined indicators of quality rehabilitation services for individuals with LEP.
      • Thirty experts reached consensus on important indicators and their feasibility.
      • Results can be used to guide policies, practices, and assessments.
      • Results inform guidelines for quality rehabilitation services for individuals with LEP.



      To obtain expert consensus on indicators of quality rehabilitation services for individuals with limited English proficiency (LEP).


      Three-round Delphi study.


      Delphi survey conducted online with 30 experts. Most experts worked in adult physical rehabilitation settings and were from Illinois (n=16), and the remaining participants were from 8 other US states or Canadian provinces.


      Experts (N=30) had a minimum of 2 publications on health care services for patients with LEP and/or a minimum of 5 years clinical experience in physical rehabilitation. Of 43 experts (11 researchers, 32 clinicians) who received the round 1 survey by e-mail, 30 returned complete responses (70% response rate). Of those, 25 completed round 2 and 24 completed round 3. Of round 1 participants, most (n =21) identified their primary professional activity as clinical, whereas the others worked in research (n =5) or education (n =4). Twenty-four were women. The median age was 43 years (range, 27-67y). Disciplines included occupational therapy (n =14), physical therapy (n =13), psychology (n=1), nursing (n=1), and medicine (n=1).


      Not applicable.

      Main Outcome Measures

      Indicators were rated on a 7-point Likert scale for importance and feasibility. Interquartile range (IQR) and 95% confidence intervals were calculated for importance and feasibility ratings. Indicators with an IQR <2 and a median importance score ≥6 were accepted as reaching consensus for importance.


      Round 1 responses were categorized into 15 structural, 13 process, and 18 outcome indicators. All 15 structural indicators reached consensus for importance; 8 were rated as feasible. All 13 process indicators reached consensus, of which 8 were deemed feasible. Sixteen outcome indicators reached consensus, of which 7 were deemed feasible.


      This Delphi study identified structural, process, and outcome indicators that can inform delivery and assessment of quality rehabilitation services for individuals with LEP. Future research should operationalize and measure these quality indicators in clinical practice.


      List of abbreviations:

      EHR (electronic health record), IQR (interquartile range), LEP (limited English proficiency), US (United States)
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Archives of Physical Medicine and Rehabilitation
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


      1. Batalova J, Zong J. Language diversity and English proficiency in the United States. Available at: Accessed December 1, 2020.

      2. US Census Bureau. Language spoken at home by ability to speak English for the population 5 years and over: ACS 5-year estimates detailed tables. Available at: Accessed April 4, 2021.

        • Al Shamsi H
        • Almutairi AG
        • Al Mashrafi S
        • Al Kalbani T
        Implications of language barriers for healthcare: a systematic review.
        Oman Med J. 2020; 35: e122
        • Flores G.
        Language barriers to health care in the United States.
        N Engl J Med. 2006; 355: 229-231
        • Mirza M
        • Harrison EA.
        Working with clients with limited English proficiency: mapping language access in occupational therapy.
        Occup Ther Health Care. 2018; 32: 105-123
      3. Pope TM. New regulations require better communication with patients who have disabilities and limited English proficiency. ASCO Post. Available at: Accessed April 4, 2021.

        • Youdelman MK.
        The medical tongue: U.S. laws and policies on language access.
        Health Aff (Millwood). 2008; 27: 424-433
        • Taira BR
        • Kim K
        • Mody N.
        Hospital and health system-level interventions to improve care for limited English proficiency patients: a systematic review.
        Jt Comm J Qual Patient Saf. 2019; 45: 446-458
        • Mirza M
        • Harrison EA
        • Roman M
        • Miller KA
        • Jacobs EA.
        Walking the talk: understanding how language barriers affect the delivery of rehabilitation services.
        Disabil Rehabil. 2020; 1: 1-14
        • Niederberger M
        • Spranger J.
        Delphi technique in health sciences: a map.
        Front Public Health. 2020; 8: 457
        • Okoli C
        • Pawlowski SD.
        The Delphi method as a research tool: an example, design considerations and applications.
        Inf Manag. 2004; 42: 15-29
        • Graham B
        • Regehr G
        • Wright JG.
        Delphi as a method to establish consensus for diagnostic criteria.
        J Clin Epidemiol. 2003; 56: 1150-1156
        • Vázquez-Ramos R
        • Leahy M
        • Estrada Hernández N
        The Delphi method in rehabilitation counseling research.
        Rehabil Couns Bull. 2007; 50: 111-118
        • Baker J
        • Lovell K
        • Harris N.
        How expert are the experts? An exploration of the concept of “expert” within Delphi panel techniques.
        Nurse Res. 2006; 14: 59-70
        • Lecours A.
        Scientific, professional and experiential validation of the model of preventive behaviours at work: protocol of a modified Delphi study.
        BMJ Open. 2020; 10e035606
        • Jorm AF.
        Using the Delphi expert consensus method in mental health research.
        Aust N Z J Psychiatry. 2015; 49: 887-897
        • Hsieh H-F
        • Shannon SE.
        Three approaches to qualitative content analysis.
        Qual Health Res. 2005; 15: 1277-1288
        • Donabedian A.
        The quality of care. How can it be assessed?.
        JAMA. 1988; 260: 1743-1748
        • Gardner K
        • Mazza D.
        Quality in general practice - definitions and frameworks.
        Aust Fam Physician. 2012; 41: 151-154
        • Berends L
        • Johnston J.
        Using multiple coders to enhance qualitative analysis: the case of interviews with consumers of drug treatment.
        Addict Res Theory. 2005; 13: 373-381
        • Fromme EK
        • Hebert RS
        • Carrese JA.
        Self-doctoring: a qualitative study of physicians with cancer.
        J Fam Pract. 2004; 53: 299-306
        • Huijben JA
        • Wiegers EJA
        • de Keizer NF
        • et al.
        Development of a quality indicator set to measure and improve quality of ICU care for patients with traumatic brain injury.
        Crit Care. 2019; 23: 95
        • Hung Y
        • Hieke S
        • Grunert KG
        • Verbeke W.
        Setting policy priorities for front-of-pack health claims and symbols in the European Union: expert consensus built by using a Delphi method.
        Nutrients. 2019; 11: 403
        • James MT
        • Pannu N
        • Barry R
        • et al.
        A modified Delphi process to identify process of care indicators for the identification, prevention and management of acute kidney injury after major surgery.
        Can J Kidney Health Dis. 2015; 2: 11
        • Schneider F
        • van Osch L
        • de Vries H.
        Identifying factors for optimal development of health-related websites: a delphi study among experts and potential future users.
        J Med Internet Res. 2012; 14: e18
        • Ogrinc G
        • Davies L
        • Goodman D
        • Batalden P
        • Davidoff F
        • Stevens D.
        SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process.
        BMJ Qual Saf. 2016; 25: 986-992
        • Taira BR
        • Orue A.
        Language assistance for limited English proficiency patients in a public ED: determining the unmet need.
        BMC Health Serv Res. 2019; 19: 56
        • Harrison EA
        • Mirza M.
        An online training to prepare occupational therapy students to work with clients with limited English proficiency and interpreters.
        J Occup Ther Educ. 2018; 2: 9
        • Teunissen E
        • Gravenhorst K
        • Dowrick C
        • et al.
        Implementing guidelines and training initiatives to improve cross-cultural communication in primary care consultations: a qualitative participatory European study.
        Int J Equity Health. 2017; 16: 32
      4. American Occupational Therapy Association. Legislation introduced to promote workforce diversity in allied health professions. Available at: Accessed December 2, 2020.

      5. Jannenga H. Bridging the PT diversity gap. Available at: Accessed December 2, 2020.

      6. Harvison N. Academic programs annual data report for academic year 2017-2018. Available at: Accessed April 4, 2021.

        • Nuciforo MA.
        Minority applicants to physical therapist education programs 2010–2012.
        Phys Ther. 2015; 95: 39-50
        • Pechak C
        • Dillon L
        • Umucu E.
        Improving patient-provider communication: evolution of a tool to assess physical therapist students’ Spanish-language proficiency.
        Health Commun. 2019; 34: 1433-1440
        • Diamond LC
        • Tuot DS
        • Karliner LS.
        The use of Spanish language skills by physicians and nurses: policy implications for teaching and testing.
        J Gen Intern Med. 2012; 27: 117-123
        • Diamond LC
        • Jacobs EA.
        Let's not contribute to disparities: the best methods for teaching clinicians how to overcome language barriers to health care.
        J Gen Intern Med. 2010; 25: 189-193
        • Baurer D
        • Yonek JC
        • Cohen AB
        • Restuccia JD
        Hasnain-Wynia R. System-level factors affecting clinicians’ perceptions and use of interpreter services in California public hospitals.
        J Immigr Minor Health. 2014; 16: 211-217
        • Diamond LC
        • Schenker Y
        • Curry L
        • Bradley EH
        • Fernandez A.
        Getting by: underuse of interpreters by resident physicians.
        J Gen Intern Med. 2009; 24: 256-262
        • Hsieh E.
        Not just “getting by”: factors influencing providers’ choice of interpreters.
        J Gen Intern Med. 2015; 30: 75-82
        • Jacobs EA
        • Leos GS
        • Rathouz PJ
        • Fu P.
        Shared networks of interpreter services, at relatively low cost, can help providers serve patients with limited English skills.
        Health Aff (Millwood). 2011; 30: 1930-1938
        • Masland MC
        • Lou C
        • Snowden L.
        Use of communication technologies to cost-effectively increase the availability of interpretation services in healthcare settings.
        Telemed J E Health. 2010; 16: 739-745
        • Kuttner R.
        Market-based failure—a second opinion on U.S. health care costs.
        N Engl J Med. 2008; 358: 549-551
        • Brandl EJ
        • Schreiter S
        • Schouler-Ocak M.
        Are trained medical interpreters worth the cost? A review of the current literature on cost and cost-effectiveness.
        J Immigr Minor Health. 2020; 22: 175-181
        • Karliner LS
        • Perez-Stable EJ
        • Gregorich SE.
        Convenient access to professional interpreters in the hospital decreases readmission rates and estimated hospital expenditures for patients with limited English proficiency.
        Med Care. 2017; 55 (299-206)
        • Novak-Zezula S
        • Schulze B
        • Karl-Trummer U
        • Krajic K
        • Pelikan JM.
        Improving interpreting in clinical communication: models of feasible practice from the European project ‘Migrant-friendly Hospitals.
        Divers Equal Health Soc Care. 2005; 2: 223-232
        • Regenstein M
        • Huang J
        • West C
        • Trott J
        • Mead H
        • Andres E
        Improving the quality of language services delivery: findings from a hospital quality improvement initiative.
        J Healthc Qual. 2012; 34: 53-63
        • Paradise RK
        • Hatch M
        • Quessa A
        • Gargano F
        • Khaliif M
        • Costa V.
        Reducing the use of ad hoc interpreters at a safety-net health care system.
        Jt Comm J Qual Patient Saf. 2019; 45: 397-405
        • Paradise RK
        • Choi YS
        • Cundiff L
        • et al.
        The language services documentation tool: documenting how patient language needs were met during clinical encounters.
        Jt Comm J Qual Patient Saf. 2014; 40: 522-528
        • Cramer GR
        • Singh SR
        • Flaherty S
        • Young GJ.
        The progress of US hospitals in addressing community health needs.
        Am J Public Health. 2017; 107: 255-261
        • Sharma A
        • Angel L
        • Bui Q.
        Patient advisory councils: giving patients a seat at the table.
        Fam Pract Manag. 2015; 22: 22-27
        • Donelan K
        • Hobrecker K
        • Schapira L
        • Mailhot JR
        • Goulart BH
        • Chabner BA.
        Medical interpreter knowledge of cancer and cancer clinical trials.
        Cancer. 2009; 115: 3283-3292
      7. Certification Commission for Healthcare Interpreters. CCHI candidate's examination handbook. Available at: Accessed April 4, 2021.

      8. National Board of Certification for Medical Interpreters. NBCMI candidate handbook. Available at: Accessed April 4, 2021.

      9. Barrett M, Steiner C, Andrews R, Kassed C, Nagamine M. Methodological issues when studying readmissions and revisits using hospital administrative data. Available at: Accessed April 4, 2021.

        • Dowbor T
        • Zerger S
        • Pedersen C
        • et al.
        Shrinking the language accessibility gap: a mixed methods evaluation of telephone interpretation services in a large, diverse urban health care system.
        Int J Equity Health. 2015; 14: 83
        • Ginde AA
        • Sullivan AF
        • Corel B
        • Caceres JA
        • Camargo CA.
        Reevaluation of the effect of mandatory interpreter legislation on use of professional interpreters for ED patients with language barriers.
        Patient Educ Couns. 2010; 81: 204-206
        • Lion KC
        • Ebel BE
        • Rafton S
        • Zhou C
        • Hencz P
        • Mangione-Smith R.
        Evaluation of a quality improvement intervention to increase use of telephonic interpretation.
        Pediatrics. 2015; 135: e709-e716
        • Schouten BC
        • Schinkel S.
        Turkish migrant GP patients’ expression of emotional cues and concerns in encounters with and without informal interpreters.
        Patient Educ Couns. 2014; 97: 23-29
        • Seale C
        • Rivas C
        • Kelly M.
        The challenge of communication in interpreted consultations in diabetes care: a mixed methods study.
        Br J Gen Pract. 2013; 63: e125-e133
        • Nápoles AM
        • Santoyo-Olsson J
        • Karliner LS
        • O'Brien H
        • Gregorich SE
        • Pérez-Stable EJ
        Clinician ratings of interpreter mediated visits in underserved primary care settings with ad hoc, in-person professional, and video conferencing modes.
        J Health Care Poor Underserved. 2010; 21: 301-317
        • Jacobs EA
        • Shepard DS
        • Suaya JA
        • Stone E-L.
        Overcoming language barriers in health care: costs and benefits of interpreter services.
        Am J Public Health. 2004; 94: 866-869