Advertisement

Feasibility of Computerized Adaptive Testing for Collection of Patient-Reported Outcomes After Inpatient Rehabilitation

Published:January 21, 2014DOI:https://doi.org/10.1016/j.apmr.2013.12.024

      Abstract

      Objective

      To evaluate the feasibility of computer adaptive testing (CAT) using an Internet or telephone interface to collect patient-reported outcomes after inpatient rehabilitation and to examine patient characteristics associated with completion of the CAT-administered measure and mode of administration.

      Design

      Prospective cohort study of patients contacted approximately 4 weeks after discharge from inpatient rehabilitation. Patients selected an Internet or telephone interface.

      Setting

      Rehabilitation hospital.

      Participants

      Patients (N=674) with diagnoses of neurologic, orthopedic, or medically complex conditions.

      Interventions

      None.

      Main Outcome Measure

      CAT version of the Community Participation Indicators (CAT-CPI).

      Results

      From an eligible pool of 3221 patients, 674 (21%) agreed to complete the CAT-CPI. Patients who agreed to complete the CAT-CPI were younger and reported slightly higher satisfaction with overall care than those who did not participate. Among these patients, 231 (34%) actually completed the CAT-CPI; 141 (61%) selected telephone administration, and 90 (39%) selected Internet administration. Decreased odds of completing the CAT-CPI were associated with black and other race; stroke, brain injury, or orthopedic and other impairments; and being a Medicaid beneficiary, whereas increased odds of completing the CAT-CPI were associated with longer length of stay and higher discharge FIM cognition measure. Decreased odds of choosing Internet administration were associated with younger age, retirement status, and being a woman, whereas increased odds of choosing Internet administration were associated with higher discharge FIM motor measure.

      Conclusions

      CAT administration by Internet and telephone has limited feasibility for collecting postrehabilitation outcomes for most rehabilitation patients, but it is feasible for a subset of patients. Providing alternative ways of answering questions helps assure that a larger proportion of patients will respond.

      Keywords

      List of abbreviations:

      CAT (computerized adaptive testing), CAT-CPI (CAT version of the Community Participation Indicators), CI (confidence interval), IVR (interactive voice response), LOS (length of stay), PRO (patient-reported outcome), OR (odds ratio)
      Feedback from patients discharged from rehabilitation settings is critical for advancing rehabilitation services and research.
      • Ware Jr., J.E.
      • Gandek B.
      • Sinclair S.J.
      • Bjorner J.B.
      Item response theory and computerized adaptive testing: implications for outcomes measurement in rehabilitation.
      Medical rehabilitation and neurologic populations can expect to live close to a normative life span because of improved medical services and technology.
      • Frankel H.
      • Coll J.
      • Charlifue S.
      • et al.
      Long-term survival in spinal cord injury: a fifty year investigation.
      However, they may experience recurring health issues that limit community participation. Therefore, tracking patients' postdischarge health status and measuring their long-term outcomes are becoming important activities for rehabilitation providers.
      • Barker R.N.
      • Amsters D.I.
      • Kendall M.D.
      • Pershouse K.J.
      • Haines T.P.
      Reliability of the clinical outcome variables scale when administered via telephone to assess mobility in people with spinal cord injury.
      • Krause J.
      • Broderick L.
      A 25-year longitudinal study of the natural course of aging after spinal cord injury.
      Patient-reported outcome (PRO) measures after discharge are typically administered via mailed questionnaires or telephone interviews with hospital staff members. Mailed surveys suffer from low response rates; interview methods are costly and time-consuming.
      • Chien T.W.
      • Wang W.C.
      • Wang H.Y.
      • Lin H.J.
      Online assessment of patients' views on hospital performances using Rasch model's KIDMAP diagram.
      • Haley S.M.
      • Siebens H.
      • Black-Schaffer R.M.
      • et al.
      Computerized adaptive testing for follow-up after discharge from inpatient rehabilitation: II. Participation outcomes.
      • Haley S.M.
      • Siebens H.
      • Coster W.J.
      • et al.
      Computerized adaptive testing for follow-up after discharge from inpatient rehabilitation. I. Activity outcomes.
      Measuring PROs using computerized adaptive testing (CAT) is an alternative data collection method that can maximize patient engagement and minimize costs.
      • Rose M.
      • Bezjak A.
      Logistics of collecting patient-reported outcomes (PROs) in clinical practice: an overview and practical examples.
      • Jones J.B.
      • Snyder C.F.
      • Wu A.W.
      Issues in the design of Internet-based systems for collecting patient-reported outcomes.
      CAT is a process of test administration in which items are selected on the basis of respondents' answers to previously administered items. This process requires that the hierarchically ordered item pools are created by item response theory methods and use an algorithm to estimate person ability and choose the next best item to administer using test specifications (eg, content coverage, desired length of precision).
      • Heinemann A.W.
      Putting outcome measurement in context: a rehabilitation psychology perspective.
      Two modes of CAT administration are available
      • Ritter P.
      • Lorig K.
      • Laurent D.
      • Matthews K.
      Internet versus mailed questionnaires: a randomized comparison.
      : automated telephone technology using interactive voice response (IVR) systems and Internet websites. Because of its ability to restrict questioning to a limited number of discriminative items, CAT offers the opportunity to be more responsive than conventional, fixed-length assessment tools and reduce respondent burden without loss of precision.
      • Cheville A.L.
      • Yost K.J.
      • Larson D.R.
      • et al.
      Performance of an item response theory-based computer adaptive test in identifying functional decline.
      In addition to improving efficiency and responsiveness, other CAT advantages over conventional methods include immediacy of feedback, communication on a common metric, and dynamic tailoring of test difficulty to the level of the individual.
      • Harniss M.
      • Amtmann D.
      • Cook D.
      • Johnson K.
      Considerations for developing interfaces for collecting patient-reported outcomes that allow the inclusion of individuals with disabilities.
      • Reeve B.B.
      Special issues for building computerized-adaptive tests for measuring patient-reported outcomes: the National Institute of Health's investment in new technology.
      PROs collected via CAT platforms are recognized as an important method to enhance patient-centered treatment and decision-making.
      • Chang C.H.
      Patient-reported outcomes measurement and management with innovative methodologies and technologies.
      However, the higher upfront costs involved with developing and maintaining the CAT, staff training on CAT administration, and the technology associated with data security and privacy may be potential barriers that restrict PROs collected via CAT platforms.
      • Snyder C.F.
      • Aaronson N.K.
      • Choucair A.K.
      • et al.
      Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations.
      During the recent decades, CAT administration has demonstrated feasibility for collecting patient outcomes in primary care settings.
      • Turner-Bowker D.M.
      • Saris-Baglama R.N.
      • DeRosa M.A.
      • Giovannetti E.R.
      • Jensen R.E.
      • Wu A.W.
      A computerized adaptive version of the SF-36 is feasible for clinic and Internet administration in adults with HIV.
      • Wu W.W.
      • Johnson R.
      • Schepp K.G.
      • Berry D.L.
      Electronic self-report symptom and quality of life for adolescent patients with cancer: a feasibility study.
      Few studies have evaluated the feasibility of PRO data collection via CAT platforms with medical rehabilitation populations. Feasibility was assessed with regard to survey completion, acceptability, time to completion, and mode of survey administration.
      • Wu W.W.
      • Johnson R.
      • Schepp K.G.
      • Berry D.L.
      Electronic self-report symptom and quality of life for adolescent patients with cancer: a feasibility study.
      The heterogeneous nature of medical rehabilitation populations requires that careful consideration be given to the optimal CAT platform for measuring PROs. Given our limited knowledge, we aimed to examine the utility of CAT as a data collection strategy for collecting postrehabilitation PROs and investigate the patient characteristics related to the completion and selection of 2 CAT platforms. We used a CAT-administered measure to assess community participation, an important long-term outcome for rehabilitation research and practice,
      • Heinemann A.W.
      Measurement of participation in rehabilitation research.
      • Heinemann A.W.
      • Lai J.S.
      • Magasi S.
      • et al.
      Measuring participation enfranchisement.
      • Whiteneck G.G.
      • Bogner J.A.
      • Heinemann A.W.
      Advancing the measurement of participation.
      with a sample of patients after discharge from inpatient rehabilitation. We also evaluated their preferences for Internet or IVR administration. We sought to answer the following 2 questions: (1) What patient characteristics are associated with the completion of a CAT-administered participation measure? (2) What patient characteristics are associated with the likelihood of selecting either Internet or telephone administration?

      Methods

       Study design

      We used a prospective cohort design to study patient characteristics associated with completing the CAT-administered measure and the selected mode of administration. Our institutional review board approved the research procedures.

       Setting and procedures

      Patients received inpatient services from a 182-bed, Midwestern, freestanding urban hospital. The hospital's outcomes management department asks patients to participate in a satisfaction survey 1 month after discharge by telephone. The hospital has a recruitment quota of 40% of discharged patients. After completing the satisfaction survey, the hospital staff members asked patients if they would complete a CAT version of the Community Participation Indicators (CAT-CPI).
      • Heinemann A.W.
      • Lai J.S.
      • Magasi S.
      • et al.
      Measuring participation enfranchisement.
      If they agreed, they were asked whether they preferred to do so on a secure website or using an IVR system. The manager of the outcomes management department agreed to invite all patients who provided self-reports on the satisfaction interview to complete the CAT-CPI between July 2009 and June 2011. Interviewers received reminders routinely to invite eligible patients and received notification and encouragement when they achieved recruitment milestones. Patients are called at different times (office and nonoffice hours and weekdays and weekends) to minimize selection bias. Patients who agreed to complete the CAT-CPI received a postcard with information about the login procedure; they received a reminder postcard if they had not logged on to the system 2 weeks after the first postcard was mailed.

       Participants

      Patients were eligible if they were aged ≥18 years, had a length of stay (LOS) >1 day, and served as their own informant during the postdischarge interview. We excluded patients who were readmitted before they were eligible to complete the satisfaction survey, were <18 years old, had a LOS ≤1 day, and for whom proxies completed the satisfaction survey. We also excluded patients who did not report correct contact information, those who were deceased, or those who did not answer the telephone when a staff member called for the postdischarge satisfaction survey. Because our selection criteria only included patients who served as their own informant, rather than included on a nonselected bias, it is likely that persons who were too cognitively impaired or unable to comprehend English-language study materials were excluded. Patients received no compensation for participation.

       Measures

      Patients completed the CAT-CPI.
      • Heinemann A.W.
      • Lai J.S.
      • Magasi S.
      • et al.
      Measuring participation enfranchisement.
      Part I of the CAT-CPI is a fixed-length set of items measuring the frequency with which people participate in 20 activities, the importance of each activity, and the level of satisfaction with the activity frequency. Respondents saw only 1 item on each screen and could not skip questions. Part II consisted of 2 item banks measuring community enfranchisement.
      • Heinemann A.W.
      • Magasi S.
      • Bode R.K.
      • et al.
      Measuring enfranchisement: importance and control of participation by people with disabilities.
      The first bank measures the importance of participation, and the second bank assesses control over participation. We used the same CAT algorithm for both banks. We set the minimum number of items to be administered for each bank at 5. Patients were able to log in to the system repeatedly to complete the CAT-CPI. We merged data from the CAT-CPI with the hospital database that included the Inpatient Rehabilitation Facility Patient Assessment Instrument

      UB Foundation Activities. The Inpatient Rehabilitation Facility - Patient Assessment Instrument (IRF-PAI) training manual. 2004. Available at: http://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/InpatientRehabFacPPS/downloads/irfpaimanual040104.pdf. Accessed May 22, 2013.

      and satisfaction with hospital care data.

       Outcome variables

      We defined 2 outcome variables for this study. The first outcome was completion of the CAT-CPI, defined as logging into the system and answering all items; we defined an unsuccessful outcome as not logging into the system or failing to complete all items. The second outcome was the mode of Community Participation Indicators administration (Internet or IVR).

       Independent variables

      We selected variables that might affect completion of a CAT-administered PRO measure based on a review of the literature and our clinical experience.
      • Barker R.N.
      • Amsters D.I.
      • Kendall M.D.
      • Pershouse K.J.
      • Haines T.P.
      Reliability of the clinical outcome variables scale when administered via telephone to assess mobility in people with spinal cord injury.
      • Haley S.M.
      • Siebens H.
      • Black-Schaffer R.M.
      • et al.
      Computerized adaptive testing for follow-up after discharge from inpatient rehabilitation: II. Participation outcomes.
      • Haley S.M.
      • Siebens H.
      • Coster W.J.
      • et al.
      Computerized adaptive testing for follow-up after discharge from inpatient rehabilitation. I. Activity outcomes.
      • Cheville A.L.
      • Yost K.J.
      • Larson D.R.
      • et al.
      Performance of an item response theory-based computer adaptive test in identifying functional decline.
      We used 2 sets of predictor variables. First, demographic characteristics included patients' age, race, marital status, premorbid vocational status, sex, diagnosis, tier group, and primary payer. Second, rehabilitation variables included inpatient rehabilitation hospital LOS, overall satisfaction with care, discharge setting, and discharge motor and cognitive status. We used the hospital's patient satisfaction instrument to measure patient satisfaction.
      • Heinemann A.W.
      • Bode R.
      • Cichowski K.
      • Kan E.
      Measuring patient satisfaction with medical rehabilitation.
      Patients rated satisfaction items on a 4-point rating scale of excellent (4), good (3), fair (2), and poor (1). We used a single item (overall satisfaction with care) because an earlier study indicated that it correlated strongly with the total score and reflected patients' global perceptions.
      • Shah P.K.
      • Heinemann A.W.
      • Manheim L.M.
      The effect of Medicare's prospective payment system on patient satisfaction: an illustration with four rehabilitation hospitals.
      We used the FIM to measure discharge functional status; we selected motor and cognitive scores because they represent 2 distinct aspects of function.
      • Heinemann A.W.
      • Linacre J.M.
      • Wright B.D.
      • Hamilton B.B.
      • Granger C.
      Measurement characteristics of the Functional Independence Measure.
      We used the Rasch model to convert the FIM scores into equal-interval measures to mitigate problems resulting from the use of ordinal scores.

      Mallinson T. Rasch analysis of repeated measure. 2011. Available at: http://www.rasch.org/rmt/rmt251b.htm. Accessed March 5, 2013.

      • Whiteneck G.
      • Gassaway J.
      SCIRehab uses practice-based evidence methodology to associate patient and treatment characteristics with outcomes.

       Statistical analyses

      We used counts, percentages, means, and SDs to describe patient characteristics; multivariate logistic regression analyses were used to examine associations between demographic, functional, and rehabilitation characteristics with the completion of the CAT-CPI and the mode of administration. We calculated odds ratios (ORs) and 95% confidence intervals (CIs) and specified statistical significance (α=.05). We used listwise deletion of cases with missing data. We examined group differences in time required for completing the 2 modes of CAT administration. We computed the number of days required for test completion by subtracting the date when all survey items were completed from the date we mailed the postcard; we compared groups computing Student t tests and Cohen d statistics. We also examined the characteristic of patients who did not log in to the system and those who dropped out. We completed all analyses with SPSS version 20.0a and Winsteps version 3.68.0.b

      Results

      Figure 1 shows the flow diagram of patients. A total of 3221 patients met the eligible criteria (table 1). The 674 patients who were asked and agreed to complete the CAT-CPI represent 21% of the patients who were interviewed during the study period, without any incentive. Because outcomes management staff forwarded the contact information only for patients who agreed to complete the CAT-CPI, we cannot distinguish patients who refused from those who were not asked to complete the CAT-CPI. We estimate that <5% of eligible patients were not invited, primarily because of a part-time staff member who was hired after the study began. Therefore, it is important to evaluate the representativeness of the patients who agreed to complete the CAT-CPI.
      Figure thumbnail gr1
      Fig 1Flow diagram of patient enrollment. *Fewer than an estimated 5% of patients were not invited to participate.
      Table 1Patient characteristics
      CharacteristicsTotal PatientsPatients Who Refused/Were Not AskedPatients Who Agreedχ2 or tPCohen d
      Small is |0.2|, medium is |0.5|, and large is |0.8|.
      95% CI for d
      Demographic characteristics
       Race (n=3077)3.189.203−.055−.142 to .032
      White2492 (81.0)1988 (81.6)504 (78.8)
      Black407 (13.2)309 (12.7)98 (15.3)
      Other race178 (5.8)140 (5.7)38 (5.9)
       Marital status (n=3196)16.623.000.083−.002 to .168
      Married1554 (48.6)1223 (48.4)331 (49.4)
      Never married518 (16.2)380 (15.0)138 (20.6)
      Widowed/separated/divorced1124 (35.2)923 (36.5)201 (30.0)
       Premorbid vocational status (n=3138)40.254.000.278.192 to .365
      Employed/sheltered/student/homemakers668 (21.3)475 (19.1)193 (29.4)
      Unemployed273 (8.7)204 (8.2)69 (10.5)
      Retired2197 (70.0)1802 (72.6)395 (60.1)
       Sex (n=3221)0.001.969.002−.083 to .087
      Male1293 (40.1)1022 (40.1)271 (40.2)
      Female1928 (59.9)1525 (59.9)403 (59.8)
       Diagnosis (n=3219)15.378.009.026−.059 to .110
      Spinal cord injury269 (8.4)196 (7.7)73 (10.8)
      Stroke428 (13.3)356 (14.0)72 (10.7)
      Brain injury218 (6.8)178 (7.0)40 (5.9)
      Other neurologic conditions206 (6.4)159 (6.2)47 (7.0)
      Orthopedic conditions1302 (40.4)1011 (39.7)291 (43.2)
      Other conditions796 (24.7)645 (25.3)151 (22.4)
       Tier (n=3221)7.639.054−.054−.139 to .031
      No comorbidity1836 (57.0)1453 (57.0)383 (56.8)
      Third tier1010 (31.4)807 (31.7)203 (30.1)
      Second tier243 (7.5)195 (7.7)48 (7.1)
      First tier132 (4.1)92 (3.6)40 (5.9)
       Primary payer (n=3221)53.611.000−.316−.401 to −.231
      Medicare2174 (67.5)1798 (70.6)376 (55.8)
      Medicaid/none161 (5.0)118 (4.6)43 (6.4)
      Commercial insurance/worker's compensation/private pay/others886 (27.5)631 (24.8)255 (37.8)
       Age (n=3221) (y)67.9±15.569.3±15.162.9±15.79.634.000.418.332 to .503
      Functional/rehabilitation characteristics
       Discharge setting (n=3221)2.748.097.072−.013 to .157
      Community discharge2847 (88.4)2239 (87.9)608 (90.2)
      Nursing home/acute hospital/died/others374 (11.6)308 (12.1)66 (9.8)
       LOS (n=3221) (d)14.29±9.6213.98±8.5915.43±12.73−2.790.005−.151−.236 to −.066
       Satisfaction with care (n=3203) (raw score)
      Satisfaction with care score ranged from 1 (poor) to 4 (excellent).
      3.64±0.553.59±0.573.80±0.47−9.756.000−.384−.470 to −.298
       Discharge FIM motor function (n=3221) (Rasch-transformed score)
      All FIM scores are Rasch transformed and ranged from 0 (lowest observed score) to 100 (highest observed score).
      63.74±12.5463.29±12.5265.46±12.50−4.001.000−.173−.258 to −.088
       Discharge FIM cognitive function (n=3221) (Rasch-transformed score)
      All FIM scores are Rasch transformed and ranged from 0 (lowest observed score) to 100 (highest observed score).
      30.20±4.9629.87±5.0431.45±4.43−8.007.000−.322−.407 to −.236
      NOTE. Values are presented as n (%), mean ± SD, or as otherwise noted.
      Small is |0.2|, medium is |0.5|, and large is |0.8|.
      Satisfaction with care score ranged from 1 (poor) to 4 (excellent).
      All FIM scores are Rasch transformed and ranged from 0 (lowest observed score) to 100 (highest observed score).
      We compared patients who agreed and those who refused or were not asked on 13 demographic, functional, and rehabilitation characteristics. Because of the large sample, relatively small differences are statistically significant (9 of 13 comparisons, P<.01). Therefore, we examined effect sizes. All comparisons yielded very small to small effects, except for 2 comparisons that were close to medium effects (Cohen d ≥|.35| and <|0.5|). Patients who refused/were not asked to complete the CAT-CPI were slightly older (69 vs 63y) and reported slightly lower satisfaction with overall care (mean, 3.6 vs 3.8 on a rating scale in which 3 means good satisfaction and 4 means excellent satisfaction) than patients who agreed to complete the CAT-CPI. The 674 (21%) eligible patients discharged from the hospital agreed to complete the CAT-administered CPI. The distribution of impairments shows that 34.4% had neurologic conditions, 43.2% had orthopedic conditions, and 22.4% had medical conditions. Of those agreeing to complete the CAT-CPI, 231 (34.3%) logged into the system and completed all items. One hundred forty-one (61%) participants chose IVR, whereas 90 (39%) chose Internet administration. Of the remaining 443 patients, 399 (90%) did not log in to the system and 44 (10%) stopped before completing all items. We compared the demographic, functional, or rehabilitation characteristics between these 2 subgroups (table 2). Among all comparisons, only age was statistically significant. Patients who stopped before completing all items were slightly older (68 vs 62y) than patients who did not log in to the system. Among the patients who did not finish the CAT, 40 (91%) chose IVR administration, whereas only 4 (9%) chose Internet administration.
      Table 2Characteristics of patients who did not complete the CAT-CPI
      CharacteristicsPatients Who Dropped OutPatients Who Did Not Log inχ2 or tPCohen d
      Small is |0.2|, medium is |0.5|, and large is |0.8|.
      95% CI for d
      Demographic characteristics
       Race (n=424)0.786.675−.139−.458 to .180
      White33 (78.6)281 (73.6)
      Black7 (16.7)69 (18.1)
      Other race2 (4.8)32 (8.4)
       Marital status (n=439)0.405.817.099−.212 to .411
      Married19 (43.2)190 (48.1)
      Never married10 (22.7)85 (21.5)
      Widowed/separated/divorced15 (34.1)120 (30.4)
       Premorbid vocational status (n=432)0.658.720.101−.220 to .423
      Employed/sheltered/student/homemakers11 (26.8)117 (29.9)
      Unemployed3 (7.3)40 (10.2)
      Retired27 (65.9)234 (59.8)
       Sex (n=443)0.471.493−.109−.421 to .202
      Male20 (45.5)160 (40.1)
      Female24 (54.5)239 (59.9)
       Diagnosis (n=443)4.195.522.200−.112 to .512
      Spinal cord injury3 (6.8)38 (9.5)
      Stroke3 (6.8)47 (11.8)
      Brain injury1 (2.3)29 (7.3)
      Other neurologic conditions4 (9.1)27 (6.8)
      Orthopedic conditions23 (52.3)163 (40.9)
      Other conditions10 (22.7)95 (23.8)
       Tier (n=443)1.472.689−.114−.425 to .197
      No comorbidity29 (65.9)229 (57.4)
      Third tier10 (22.7)117 (29.3)
      Second tier2 (4.5)28 (7.0)
      First tier3 (6.8)25 (6.3)
       Primary payer (n=443)1.322.516−.208−.593 to .176
      Medicare27 (61.4)216 (54.1)
      Medicaid/none2 (4.5)35 (8.8)
      Commercial insurance/worker's compensation/private pay/others15 (34.1)148 (37.1)
       Age (n=443) (y)2.127.034.339.026 to .650
      Functional/rehabilitation characteristics67.7±13.662.1±16.8
       Discharge setting (n=443)0.329.566.091−.220 to .403
      Community discharge38 (86.4)356 (89.2)
      Nursing home/acute hospital/died/others6 (13.6)43 (10.8)
       LOS (n=443) (d)14.3±7.815.0±12.7−0.334.738−.053−.364 to .258
       Satisfaction with care (n=439) (raw score)
      Satisfaction with care score ranged from 1 (poor) to 4 (excellent).
      3.84±0.373.78±0.490.830.407.125−.186 to .437
       Discharge FIM motor function (n=443) (Rasch-transformed score)
      All FIM scores are Rasch transformed and ranged from 0 (lowest observed score) to 100 (highest observed score).
      63.93±10.7065.43±13.17−0.729.467−.116−.427 to .196
       Discharge FIM cognitive function (n=443) (Rasch-transformed score)
      All FIM scores are Rasch transformed and ranged from 0 (lowest observed score) to 100 (highest observed score).
      30.61±3.8431.20±4.57−0.820.413−.130−.442 to .181
      NOTE. Values are presented as n (%), mean ± SD, or as otherwise noted.
      Small is |0.2|, medium is |0.5|, and large is |0.8|.
      Satisfaction with care score ranged from 1 (poor) to 4 (excellent).
      All FIM scores are Rasch transformed and ranged from 0 (lowest observed score) to 100 (highest observed score).

       Completion of the CAT-CPI

      We modeled patients' completion of the CAT-CPI using 13 demographic, functional, and rehabilitation variables to predict the likelihood of CAT-CPI completion.
      The omnibus test for the logistic regression model was statistically significant [χ2(23)=63.398, p<0.001]. The Nagelkerke R2 was .14. The Hosmer-Lemeshow goodness-of-fit test was not statistically significant [χ2(8)=15.289, p=0.054]. This P value suggests that the model fit is marginal, indicating that the results may not be replicable. We entered demographic characteristics in the first step. Compared with white patients, black patients had a 57% reduced likelihood of completing the CAT-CPI (OR=.43; 95% CI, .25–.76); patients from other racial backgrounds had an 87% reduced likelihood (OR=.13; 95% CI, .04–.40). Compared with patients with spinal cord injuries, patients with stroke had a 55% reduced likelihood of completing the CAT-CPI (OR=.45; 95% CI, .21–.98). Patients with brain injury had an 81% reduced likelihood of completing the CAT-CPI (OR=.19; 95% CI, .07–.52). Patients with orthopedic conditions had a 49% reduced likelihood of completing the CAT-CPI (OR=.51; 95% CI, .27–.97). Patients with other impairments had a 64% reduced likelihood (OR=.36; 95% CI, .18–.73) of completing the CAT-CPI. Medicaid beneficiaries had a 74% reduced likelihood of completing the measure (OR=.26; 95% CI, .09–.72) compared with Medicare beneficiaries.
      In the second step, we added 5 functional and rehabilitation variables as predictors. Black patients, patients from other racial backgrounds, patients with other impairment, and Medicaid beneficiaries remained less likely to complete the CAT-CPI; LOS and discharge FIM cognition measure were associated with a greater likelihood of completing the CAT-CPI (table 3). A 1-unit increase in rehabilitation LOS (OR=1.02; 95% CI, 1.001–1.05) was associated with 2% increased odds of completing the CAT-CPI. A 1-unit increase in discharge FIM cognition score (OR=1.07; 95% CI, 1.01–1.13) was associated with a 7% increased odds of completing the CAT-CPI. The wide range of 95% CIs around the ORs appeared in most of these significant relations, suggesting that these ORs lack precision; these relations should be interpreted cautiously.
      Table 3Patient characteristics associated with completion of the CAT-CPI (n=614)
      Independent VariablesStep 1Step 2
      OR (95% CI)POR (95% CI)P
      Demographic characteristics
       Race
      White1 (reference)1 (reference)
      Black0.434 (0.246–0.763).0040.388 (0.218–0.692).001
      Other race0.126 (0.040–0.395).0000.096 (0.029–0.321).000
       Marital status
      Married1 (reference)1 (reference)
      Never married1.307 (0.750–2.278).3451.434 (0.809–2.543).218
      Widowed/separated/divorced1.012 (0.665–1.538).9571.018 (0.664–1.561).934
       Premorbid vocational status
      Employed/sheltered/student/homemakers1 (reference)1 (reference)
      Unemployed1.871 (0.932–3.757).0781.861 (0.916–3.781).086
      Retired1.269 (0.693–2.323).4401.392 (0.748–2.588).297
       Sex
      Male1 (reference)1 (reference)
      Female1.282 (0.868–1.892).2121.300 (0.876–1.929).192
       Diagnosis
      Spinal cord injury1 (reference)1 (reference)
      Stroke0.452 (0.208–0.983).0450.684 (0.293–1.599).381
      Brain injury0.185 (0.066–0.521).0010.326 (0.104–1.025).055
      Other neurologic conditions0.417 (0.169–1.029).0580.519 (0.204–1.321).169
      Orthopedic conditions0.509 (0.267–0.968).0400.595 (0.284–1.244).168
      Other conditions0.362 (0.180–0.727).0040.448 (0.209–0.961).039
       Tier
      No comorbidity1 (reference)1 (reference)
      Third tier1.403 (0.943–2.086).0951.352 (0.897–2.037).150
      Second tier1.287 (0.588–2.816).5281.131 (0.510–2.511).762
      First tier1.856 (0.800–4.306).1502.200 (0.912–5.311).079
       Primary payer
      Medicare1 (reference)1 (reference)
      Medicaid/none0.256 (0.090–0.724).0100.294 (0.103–0.837).022
      Commercial insurance/worker's compensation/private pay/others1.214 (0.683–2.160).5091.261 (0.699–2.273).441
       Age
      These variables are allowed to enter the regression models as continuous data.
      0.994 (0.977–1.013).5481.001 (0.982–1.020).952
      Functional and rehabilitation characteristics
       Discharge setting
      Community dischargeNA1 (reference)
      Nursing home/acute hospital/died/othersNA0.801 (0.417–1.535).503
       LOS
      These variables are allowed to enter the regression models as continuous data.
      NA1.023 (1.001–1.046).041
       Satisfaction with care
      These variables are allowed to enter the regression models as continuous data.
      NA1.133 (0.760–1.688).541
       Discharge FIM motor function
      These variables are allowed to enter the regression models as continuous data.
      NA1.011 (0.993–1.030).244
       Discharge FIM cognitive function
      These variables are allowed to enter the regression models as continuous data.
      NA1.067 (1.010–1.126).020
      NOTE. Variables are allowed to enter the regression models as categorical data unless otherwise noted.
      Abbreviation: NA, not applicable.
      These variables are allowed to enter the regression models as continuous data.

       Modality selection

      We modeled the selection of Internet versus IVR modality for patients who completed the CAT-CPI using the same set of predictors. The omnibus test for the logistic regression model was statistically significant (χ223,207=51.73, P=.001). The Nagelkerke R2 was .30, indicating that 30% of the variance in choosing Internet administration is explained by the predictors. The Hosmer-Lemeshow goodness-of-fit test was not statistically significant (χ28,207=4.133, P=.845), indicating that the model fits the data well. In the first step, older patients were 4% less likely (OR=.96; 95% CI, .93–.999) to choose Internet administration. Compared with patients who worked before illness, retired patients were 67% less likely (OR=.33; 95% CI, .12–.90) to choose Internet administration. Women were 54% less likely than men to choose Internet administration (OR=.46; 95% CI, .23–.93).
      After we entered the functional and rehabilitation variables, retired patients remained less likely to select Internet administration. Race and discharge FIM motor measure were related to mode of administration (table 4). Black patients had a 77% lower likelihood of choosing Internet administration (OR=.23; 95% CI, .06–.92) than white patients. A 1-unit increase in discharge FIM motor scores (OR=1.04; 95% CI, 1.002–1.09) was associated with 4% increased odds of choosing Internet administration.
      Table 4Patient characteristics associated with the choice of Internet administration (n=207)
      Independent VariablesStep 1Step 2
      OR (95% CI)POR (95% CI)P
      Demographic characteristics
       Race
      White1 (reference)1 (reference)
      Black0.289 (0.077–1.092).0670.231 (0.058–0.915).037
      Other race0.608 (0.047–7.811).7020.417 (0.029–6.085).523
       Marital status
      Married1 (reference)1 (reference)
      Never married0.697 (0.270–1.796).4550.799 (0.297–2.149).657
      Widowed/separated/divorced0.487 (0.213–1.115).0890.474 (0.202–1.110).085
       Premorbid vocational status
      Employed/sheltered/student/homemakers1 (reference)1 (reference)
      Unemployed1.135 (0.354–3.641).8311.435 (0.418–4.922).566
      Retired0.328 (0.120–0.896).0300.350 (0.122–1.001).050
       Sex
      Male1 (reference)1 (reference)
      Female0.459 (0.226–0.930).0310.483 (0.231–1.009).053
       Diagnosis
      Spinal cord injury1 (reference)1 (reference)
      Stroke0.847 (0.214–3.356).8141.321 (0.263–6.650).735
      Brain injury0.309 (0.044–2.161).2360.713 (0.049–10.430).805
      Other neurologic conditions1.343 (0.276–6.527).7141.312 (0.251–6.855).747
      Orthopedic conditions1.136 (0.365–3.532).8250.803 (0.215–2.999).744
      Other conditions1.667 (0.458–6.071).4391.462 (0.349–6.122).603
       Tier
      No comorbidity1 (reference)1 (reference)
      Third tier1.744 (0.843–3.606).1332.027 (0.937–4.383).073
      Second tier1.035 (0.241–4.441).9631.139 (0.258–5.029).864
      First tier1.086 (0.260–4.535).9102.615 (0.481–14.200).266
       Primary payer
      Medicare1 (reference)1 (reference)
      Medicaid/none0.493 (0.045–5.441).5640.870 (0.072–10.438).912
      Commercial insurance/worker's compensation/private pay/others0.657 (0.247–1.744).3990.627 (0.225–1.744).371
       Age
      These variables are allowed to enter the regression models as continuous data.
      0.963 (0.929–0.999).0420.968 (0.932–1.005).086
      Functional/rehabilitation characteristics
       Discharge setting
       Community dischargeNA1 (reference)
       Nursing home/acute hospital/died/othersNA0.385 (0.084–1.764).219
       LOS
      These variables are allowed to enter the regression models as continuous data.
      NA1.005 (0.973–1.038).776
       Satisfaction with care
      These variables are allowed to enter the regression models as continuous data.
      NA0.802 (0.381–1.689).561
       Discharge FIM motor function
      These variables are allowed to enter the regression models as continuous data.
      NA1.043 (1.002–1.085).040
       Discharge FIM cognitive function
      These variables are allowed to enter the regression models as continuous data.
      NA1.087 (0.963–1.227).176
      NOTE. Variables are allowed to enter the regression models as categorical data unless otherwise noted.
      Abbreviation: NA, not applicable.
      These variables are allowed to enter the regression models as continuous data.

       Secondary outcomes

      Patients who chose Internet administration, since the first postcard was mailed, took longer to log into the system and complete the participation measure (11.9±9.8d) than patients who chose IVR (9.0±9.8d; t226,228=−2.157, P=.032). Cohen d was .29, indicating a small effect size. Although more patients chose IVR administration, a smaller proportion of this group completed the CAT (141 out of 181 patients or 78%) than patients who selected the Internet option (90 out of 94 patients or 96%; χ21,275=14.658, P<.001).

      Discussion

      The purpose of this study was to examine the feasibility of a CAT-administered PRO for data collection after rehabilitation hospitalization and examine patient, functional, and rehabilitation characteristics associated with completion of the CAT-administered PRO and the mode of administration selected. We examined these questions prospectively in a cohort of patients who were discharged from inpatient rehabilitation. Twenty-one percent of the eligible patients agreed to complete the CAT-administered PRO without an incentive; among these patients, 34% completed the CAT-administered PRO. Therefore, only 7% of the eligible patients actually completed the CAT-administered PRO. Those agreeing and declining to complete the CAT-administered PRO were similar in most demographic, rehabilitation, and functional characteristics, except that older patients and those reporting slightly lower overall satisfaction were less likely to agree to complete the CAT-administered PRO. Although the study design does not address why patients declined, staff feedback suggests that feeling burdened by voluntary opportunities to provide postdischarge health outcomes was not a priority.
      The low response rate suggests that the use of the CAT platform for collecting postrehabilitation PROs may not be a viable option for most rehabilitation populations without an incentive. Earlier studies
      • Blackstone M.M.
      • Wiebe D.J.
      • Mollen C.J.
      • Kalra A.
      • Fein J.A.
      Feasibility of an interactive voice response tool for adolescent assault victims.
      • Forster A.J.
      • Boyle L.
      • Shojania K.G.
      • Feasby T.E.
      • van Walraven C.
      Identifying patients with post-discharge care problems using an interactive voice response system.
      • Forster A.J.
      • van Walraven C.
      Using an interactive voice response system to improve patient safety following hospital discharge.
      that found CAT-administered PROs are feasible in many clinical settings; however, the use of a CAT platform for collecting PROs after rehabilitation hospitalization may pose greater challenges, perhaps because of the heterogeneous nature of rehabilitation patient populations. Despite these challenges, our findings may support the feasibility of CAT-administered PROs for postrehabilitation data collection in a subset of patients, including those with higher cognitive function at discharge, longer LOSs, spinal cord injury, nonminority status, and nonpoverty level incomes. Focusing on patients with these characteristics allows for a more representative postdischarge rehabilitation sample. Further research is needed to support such a conclusion.
      Prior rehabilitation studies regarding CAT applications have focused primarily on the evaluation of instruments' responsiveness to change and respondent burden among rehabilitation patients.
      • Haley S.M.
      • Siebens H.
      • Black-Schaffer R.M.
      • et al.
      Computerized adaptive testing for follow-up after discharge from inpatient rehabilitation: II. Participation outcomes.
      • Haley S.M.
      • Siebens H.
      • Coster W.J.
      • et al.
      Computerized adaptive testing for follow-up after discharge from inpatient rehabilitation. I. Activity outcomes.
      • Cheville A.L.
      • Yost K.J.
      • Larson D.R.
      • et al.
      Performance of an item response theory-based computer adaptive test in identifying functional decline.
      • Jette A.M.
      • Haley S.M.
      • Tao W.
      • et al.
      Prospective evaluation of the AM-PAC-CAT in outpatient rehabilitation settings.
      This investigation extends prior studies by examining patient demographic, rehabilitation, and functional characteristics associated with the completion of a CAT-administered PRO. Lower completion rates were associated with minority status, diagnosis other than spinal cord injury, Medicaid insurance, shorter LOSs, and lower cognitive function at discharge. Medicaid insurance is probably a proxy for low-income status. These associations underscore the risk of selection bias in the exclusive use of Internet and IVR for data collection. Although access to the telecommunication technologies has increased (eg, 77% of the U.S. population has Internet access),

      Internet World Stats. United States of America internet usage and broadband usage. 2010. Available at: http://www.internetworldstats.com/am/us.htm. Accessed May 6, 2013.

      Americans from racial and ethnic minorities (51%) are less likely to own a desktop computer than whites (65%).

      Smith A. Technology trends among people of color. 2010. Available at: http://www.pewinternet.org/Commentary/2010/September/Technology-Trends-Among-People-of-Color.aspx. Accessed May 6, 2013.

      Similarly, Americans from racial and ethnic minorities (56%) are less likely than whites (67%) to use the Internet or broadband technologies.

      Smith A. Home broadband 2010. 2010. Available at: http://www.pewinternet.org/Reports/2010/Home-Broadband-2010.aspx. Accessed December 6, 2013.

      Americans in higher income households (≥$75,000) have greater access to the Internet and are more likely to own telecommunication devices than those in lower income households (<$30,000).

      Jansen J. Use of the internet in higher-income households. 2010. Available at: http://pewinternet.org/Reports/2010/Better-off-households.aspx. Accessed May 6, 2013.

      Special effort is required to assure that lower income patients are able to access technology used in CAT-administered instruments. IVR is an important option given that more respondents in this study selected the IVR option and that telephone access is greater than Internet access.
      • Corkrey R.
      • Parkinson L.
      Interactive voice response: review of studies 1989-2000.
      • Lee H.
      • Friedman M.E.
      • Cukor P.
      • Ahern D.
      Interactive voice response system (IVRS) in health care services.
      • Abu-Hasaballah K.
      • James A.
      • Aseltine Jr., R.H.
      Lessons and pitfalls of interactive voice response in medical research.
      Among those who completed the CAT-CPI, most (61%) selected and completed the IVR option; patients who completed the CAT-CPI using the IVR were 3 days quicker, on average, than those using the Internet. Preference for IVR was associated with older age, retirement status, being a woman, minority status, and lower motor function at discharge. Although more patients chose IVR administration, a smaller proportion of this group who started the CAT actually completed it (78%) compared with patients who selected the Internet option (96%). It may be that the IVR administration imposes greater challenges or is more burdensome because of the time required. Implementing routine PRO measurement involves a number of considerations, including selection of the appropriate patients, settings, timing of assessment, and the optimal mode of administration.
      • Rose M.
      • Bezjak A.
      Logistics of collecting patient-reported outcomes (PROs) in clinical practice: an overview and practical examples.
      • Chang C.H.
      Patient-reported outcomes measurement and management with innovative methodologies and technologies.
      • Snyder C.F.
      • Aaronson N.K.
      • Choucair A.K.
      • et al.
      Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations.
      Offering both IVR and Internet modalities may help minimize biased sampling and increase patient participation. A qualitative or mixed-methods approach could help us understand and address the reasons for not completing the CAT.
      Although it is feasible to use a CAT platform for collecting postrehabilitation PROs for a select group of patients, their responses may be biased because of age and satisfaction with care. Patients who agreed to complete the CAT-administered PRO reported higher satisfaction with overall care, highlighting that the rehabilitation care experience may affect patients' motivation to complete and provide feedback after discharge.

       Study limitations

      Generalizing the study conclusions is limited by several factors. Patients were recruited from 1 rehabilitation hospital in a metropolitan area; therefore, results may not generalize to persons living in other regions. This study protocol invited only the patients who provided self-reports on the telephone-based satisfaction survey; patients with severe cognitive and motor function, those with communication disabilities, and those without telephone service were excluded from this study. A major difficulty with self-administered PROs is the cognitive and communication skills required to answer items independently and in a reliable manner. Data collection methods that require cognitive and communication skills may disenfranchise individuals and create selection bias that limits the usefulness of study data.
      • Quatrano L.A.
      • Cruz T.H.
      Future of outcomes measurement: impact on research in medical rehabilitation and neurologic populations.
      Language proficiency is one of the major predictors of technology use, even controlling for other demographic factors.

      Smith A. Technology trends among people of color. 2010. Available at: http://www.pewinternet.org/Commentary/2010/September/Technology-Trends-Among-People-of-Color.aspx. Accessed May 6, 2013.

      Future studies should develop a multilanguage CAT platform for collecting postrehabilitation PROs to permit broader accessibility, particular for culturally diverse patient populations who speak little English.
      • Chang C.H.
      Patient-reported outcomes measurement and management with innovative methodologies and technologies.
      Patients might have received assistance to complete the CAT-CPI, which may influence the results. Asking patients to complete the CAT-administered measure immediately after they completed the hospital's patient satisfaction instrument may create bias. Response compliance may be enhanced if CAT-administered PROs are administered in a routine postrehabilitation outcome measurement protocol. Clear guidelines are needed in the use of CATs. The low response rate limits generalization of the findings to the entire rehabilitation population. Suitable incentives may enhance patients' motivation for participation in research and clinical practice initiatives, thereby providing greater opportunities for outcome assessment after hospitalization. Results on patient characteristics associated with completion of the CAT-administered measure should be interpreted cautiously. Effective incentives are needed to increase the response rate and the representativeness of the sample. We did not document whether the nonrespondents (ie, those who did not log in to the system) would have chosen the same mode of administration again. Future studies should evaluate CAT-administered participation measures in routine, postrehabilitation quality monitoring protocols and examine which patient characteristics are associated with the selection and completion of different modes of CAT administration. Reasons for declining to complete CAT-administered PRO measures should also be studied.

      Conclusions

      This study supports the feasibility of CAT-administered PROs by Internet or IVR administration for collection of postrehabilitation PROs for a specific subset of patients. PRO completion was associated with cognitive function, LOS, payer, diagnosis, and minority status. Modality preference was associated with motor function, retirement status, and minority status. Providing alternative ways of answering questions helps assure that a larger proportion of patients have the ability to respond.

      Suppliers

      Acknowledgments

      We thank Holly DeMark Neumann, MPPA, at the Rehabilitation Institute of Chicago for research coordination, and Chih-Hung Chang, PhD, at the Rehabilitation Institute of Chicago, for insightful comments on an earlier draft of this manuscript.

      References

        • Ware Jr., J.E.
        • Gandek B.
        • Sinclair S.J.
        • Bjorner J.B.
        Item response theory and computerized adaptive testing: implications for outcomes measurement in rehabilitation.
        Rehabil Psychol. 2005; 50: 71-78
        • Frankel H.
        • Coll J.
        • Charlifue S.
        • et al.
        Long-term survival in spinal cord injury: a fifty year investigation.
        Spinal Cord. 1998; 36: 266-274
        • Barker R.N.
        • Amsters D.I.
        • Kendall M.D.
        • Pershouse K.J.
        • Haines T.P.
        Reliability of the clinical outcome variables scale when administered via telephone to assess mobility in people with spinal cord injury.
        Arch Phys Med Rehabil. 2007; 88: 632-637
        • Krause J.
        • Broderick L.
        A 25-year longitudinal study of the natural course of aging after spinal cord injury.
        Spinal Cord. 2005; 43: 349-356
        • Chien T.W.
        • Wang W.C.
        • Wang H.Y.
        • Lin H.J.
        Online assessment of patients' views on hospital performances using Rasch model's KIDMAP diagram.
        BMC Health Serv Res. 2009; 9: 1-11
        • Haley S.M.
        • Siebens H.
        • Black-Schaffer R.M.
        • et al.
        Computerized adaptive testing for follow-up after discharge from inpatient rehabilitation: II. Participation outcomes.
        Arch Phys Med Rehabil. 2008; 89: 275-283
        • Haley S.M.
        • Siebens H.
        • Coster W.J.
        • et al.
        Computerized adaptive testing for follow-up after discharge from inpatient rehabilitation. I. Activity outcomes.
        Arch Phys Med Rehabil. 2006; 87: 1033-1042
        • Rose M.
        • Bezjak A.
        Logistics of collecting patient-reported outcomes (PROs) in clinical practice: an overview and practical examples.
        Qual Life Res. 2009; 18: 125-136
        • Jones J.B.
        • Snyder C.F.
        • Wu A.W.
        Issues in the design of Internet-based systems for collecting patient-reported outcomes.
        Qual Life Res. 2007; 16: 1407-1417
        • Heinemann A.W.
        Putting outcome measurement in context: a rehabilitation psychology perspective.
        Rehabil Psychol. 2005; 50: 6-14
        • Ritter P.
        • Lorig K.
        • Laurent D.
        • Matthews K.
        Internet versus mailed questionnaires: a randomized comparison.
        J Med Internet Res. 2004; 6: e29
        • Cheville A.L.
        • Yost K.J.
        • Larson D.R.
        • et al.
        Performance of an item response theory-based computer adaptive test in identifying functional decline.
        Arch Phys Med Rehabil. 2012; 93: 1153-1160
        • Harniss M.
        • Amtmann D.
        • Cook D.
        • Johnson K.
        Considerations for developing interfaces for collecting patient-reported outcomes that allow the inclusion of individuals with disabilities.
        Med Care. 2007; 45: S48-S54
        • Reeve B.B.
        Special issues for building computerized-adaptive tests for measuring patient-reported outcomes: the National Institute of Health's investment in new technology.
        Med Care. 2006; 44: S198-S204
        • Chang C.H.
        Patient-reported outcomes measurement and management with innovative methodologies and technologies.
        Qual Life Res. 2007; 16: 157-166
        • Snyder C.F.
        • Aaronson N.K.
        • Choucair A.K.
        • et al.
        Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations.
        Qual Life Res. 2012; 21: 1305-1314
        • Turner-Bowker D.M.
        • Saris-Baglama R.N.
        • DeRosa M.A.
        • Giovannetti E.R.
        • Jensen R.E.
        • Wu A.W.
        A computerized adaptive version of the SF-36 is feasible for clinic and Internet administration in adults with HIV.
        AIDS Care. 2012; 24: 886-896
        • Wu W.W.
        • Johnson R.
        • Schepp K.G.
        • Berry D.L.
        Electronic self-report symptom and quality of life for adolescent patients with cancer: a feasibility study.
        Cancer Nurs. 2011; 34: 479-486
        • Heinemann A.W.
        Measurement of participation in rehabilitation research.
        Arch Phys Med Rehabil. 2010; 91: S1-S4
        • Heinemann A.W.
        • Lai J.S.
        • Magasi S.
        • et al.
        Measuring participation enfranchisement.
        Arch Phys Med Rehabil. 2011; 92: 564-571
        • Whiteneck G.G.
        • Bogner J.A.
        • Heinemann A.W.
        Advancing the measurement of participation.
        Arch Phys Med Rehabil. 2011; 92: 540-541
        • Heinemann A.W.
        • Magasi S.
        • Bode R.K.
        • et al.
        Measuring enfranchisement: importance and control of participation by people with disabilities.
        Arch Phys Med Rehabil. 2013; 94: 2157-2165
      1. UB Foundation Activities. The Inpatient Rehabilitation Facility - Patient Assessment Instrument (IRF-PAI) training manual. 2004. Available at: http://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/InpatientRehabFacPPS/downloads/irfpaimanual040104.pdf. Accessed May 22, 2013.

        • Heinemann A.W.
        • Bode R.
        • Cichowski K.
        • Kan E.
        Measuring patient satisfaction with medical rehabilitation.
        J Rehabil Outcome Meas. 1997; 1: 52-65
        • Shah P.K.
        • Heinemann A.W.
        • Manheim L.M.
        The effect of Medicare's prospective payment system on patient satisfaction: an illustration with four rehabilitation hospitals.
        Qual Life Res. 2007; 86: 169-175
        • Heinemann A.W.
        • Linacre J.M.
        • Wright B.D.
        • Hamilton B.B.
        • Granger C.
        Measurement characteristics of the Functional Independence Measure.
        Top Stroke Rehabil. 1994; 1: 1-15
      2. Mallinson T. Rasch analysis of repeated measure. 2011. Available at: http://www.rasch.org/rmt/rmt251b.htm. Accessed March 5, 2013.

        • Whiteneck G.
        • Gassaway J.
        SCIRehab uses practice-based evidence methodology to associate patient and treatment characteristics with outcomes.
        Arch Phys Med Rehabil. 2013; 94: S67-S74
        • Blackstone M.M.
        • Wiebe D.J.
        • Mollen C.J.
        • Kalra A.
        • Fein J.A.
        Feasibility of an interactive voice response tool for adolescent assault victims.
        Acad Emerg Med. 2009; 16: 956-962
        • Forster A.J.
        • Boyle L.
        • Shojania K.G.
        • Feasby T.E.
        • van Walraven C.
        Identifying patients with post-discharge care problems using an interactive voice response system.
        J Gen Intern Med. 2009; 24: 520-525
        • Forster A.J.
        • van Walraven C.
        Using an interactive voice response system to improve patient safety following hospital discharge.
        J Eval Clin Pract. 2007; 13: 346-351
        • Jette A.M.
        • Haley S.M.
        • Tao W.
        • et al.
        Prospective evaluation of the AM-PAC-CAT in outpatient rehabilitation settings.
        Phys Ther. 2007; 87: 385-398
      3. Internet World Stats. United States of America internet usage and broadband usage. 2010. Available at: http://www.internetworldstats.com/am/us.htm. Accessed May 6, 2013.

      4. Smith A. Technology trends among people of color. 2010. Available at: http://www.pewinternet.org/Commentary/2010/September/Technology-Trends-Among-People-of-Color.aspx. Accessed May 6, 2013.

      5. Smith A. Home broadband 2010. 2010. Available at: http://www.pewinternet.org/Reports/2010/Home-Broadband-2010.aspx. Accessed December 6, 2013.

      6. Jansen J. Use of the internet in higher-income households. 2010. Available at: http://pewinternet.org/Reports/2010/Better-off-households.aspx. Accessed May 6, 2013.

        • Corkrey R.
        • Parkinson L.
        Interactive voice response: review of studies 1989-2000.
        Behav Res Methods Instrum Comput. 2002; 34: 342-353
        • Lee H.
        • Friedman M.E.
        • Cukor P.
        • Ahern D.
        Interactive voice response system (IVRS) in health care services.
        Nurs Outlook. 2003; 51: 277-283
        • Abu-Hasaballah K.
        • James A.
        • Aseltine Jr., R.H.
        Lessons and pitfalls of interactive voice response in medical research.
        Contemp Clin Trials. 2007; 28: 593-602
        • Quatrano L.A.
        • Cruz T.H.
        Future of outcomes measurement: impact on research in medical rehabilitation and neurologic populations.
        Arch Phys Med Rehabil. 2011; 92: S7-11