If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Center for Rehabilitation Outcomes Research, Rehabilitation Institute of Chicago, Chicago, ILDepartment of Physical Medicine and Rehabilitation, Northwestern University Feinberg School of Medicine, Chicago, IL
To evaluate the feasibility of computer adaptive testing (CAT) using an Internet or telephone interface to collect patient-reported outcomes after inpatient rehabilitation and to examine patient characteristics associated with completion of the CAT-administered measure and mode of administration.
Prospective cohort study of patients contacted approximately 4 weeks after discharge from inpatient rehabilitation. Patients selected an Internet or telephone interface.
Patients (N=674) with diagnoses of neurologic, orthopedic, or medically complex conditions.
Main Outcome Measure
CAT version of the Community Participation Indicators (CAT-CPI).
From an eligible pool of 3221 patients, 674 (21%) agreed to complete the CAT-CPI. Patients who agreed to complete the CAT-CPI were younger and reported slightly higher satisfaction with overall care than those who did not participate. Among these patients, 231 (34%) actually completed the CAT-CPI; 141 (61%) selected telephone administration, and 90 (39%) selected Internet administration. Decreased odds of completing the CAT-CPI were associated with black and other race; stroke, brain injury, or orthopedic and other impairments; and being a Medicaid beneficiary, whereas increased odds of completing the CAT-CPI were associated with longer length of stay and higher discharge FIM cognition measure. Decreased odds of choosing Internet administration were associated with younger age, retirement status, and being a woman, whereas increased odds of choosing Internet administration were associated with higher discharge FIM motor measure.
CAT administration by Internet and telephone has limited feasibility for collecting postrehabilitation outcomes for most rehabilitation patients, but it is feasible for a subset of patients. Providing alternative ways of answering questions helps assure that a larger proportion of patients will respond.
However, they may experience recurring health issues that limit community participation. Therefore, tracking patients' postdischarge health status and measuring their long-term outcomes are becoming important activities for rehabilitation providers.
Patient-reported outcome (PRO) measures after discharge are typically administered via mailed questionnaires or telephone interviews with hospital staff members. Mailed surveys suffer from low response rates; interview methods are costly and time-consuming.
CAT is a process of test administration in which items are selected on the basis of respondents' answers to previously administered items. This process requires that the hierarchically ordered item pools are created by item response theory methods and use an algorithm to estimate person ability and choose the next best item to administer using test specifications (eg, content coverage, desired length of precision).
: automated telephone technology using interactive voice response (IVR) systems and Internet websites. Because of its ability to restrict questioning to a limited number of discriminative items, CAT offers the opportunity to be more responsive than conventional, fixed-length assessment tools and reduce respondent burden without loss of precision.
In addition to improving efficiency and responsiveness, other CAT advantages over conventional methods include immediacy of feedback, communication on a common metric, and dynamic tailoring of test difficulty to the level of the individual.
However, the higher upfront costs involved with developing and maintaining the CAT, staff training on CAT administration, and the technology associated with data security and privacy may be potential barriers that restrict PROs collected via CAT platforms.
Few studies have evaluated the feasibility of PRO data collection via CAT platforms with medical rehabilitation populations. Feasibility was assessed with regard to survey completion, acceptability, time to completion, and mode of survey administration.
The heterogeneous nature of medical rehabilitation populations requires that careful consideration be given to the optimal CAT platform for measuring PROs. Given our limited knowledge, we aimed to examine the utility of CAT as a data collection strategy for collecting postrehabilitation PROs and investigate the patient characteristics related to the completion and selection of 2 CAT platforms. We used a CAT-administered measure to assess community participation, an important long-term outcome for rehabilitation research and practice,
with a sample of patients after discharge from inpatient rehabilitation. We also evaluated their preferences for Internet or IVR administration. We sought to answer the following 2 questions: (1) What patient characteristics are associated with the completion of a CAT-administered participation measure? (2) What patient characteristics are associated with the likelihood of selecting either Internet or telephone administration?
We used a prospective cohort design to study patient characteristics associated with completing the CAT-administered measure and the selected mode of administration. Our institutional review board approved the research procedures.
Setting and procedures
Patients received inpatient services from a 182-bed, Midwestern, freestanding urban hospital. The hospital's outcomes management department asks patients to participate in a satisfaction survey 1 month after discharge by telephone. The hospital has a recruitment quota of 40% of discharged patients. After completing the satisfaction survey, the hospital staff members asked patients if they would complete a CAT version of the Community Participation Indicators (CAT-CPI).
If they agreed, they were asked whether they preferred to do so on a secure website or using an IVR system. The manager of the outcomes management department agreed to invite all patients who provided self-reports on the satisfaction interview to complete the CAT-CPI between July 2009 and June 2011. Interviewers received reminders routinely to invite eligible patients and received notification and encouragement when they achieved recruitment milestones. Patients are called at different times (office and nonoffice hours and weekdays and weekends) to minimize selection bias. Patients who agreed to complete the CAT-CPI received a postcard with information about the login procedure; they received a reminder postcard if they had not logged on to the system 2 weeks after the first postcard was mailed.
Patients were eligible if they were aged ≥18 years, had a length of stay (LOS) >1 day, and served as their own informant during the postdischarge interview. We excluded patients who were readmitted before they were eligible to complete the satisfaction survey, were <18 years old, had a LOS ≤1 day, and for whom proxies completed the satisfaction survey. We also excluded patients who did not report correct contact information, those who were deceased, or those who did not answer the telephone when a staff member called for the postdischarge satisfaction survey. Because our selection criteria only included patients who served as their own informant, rather than included on a nonselected bias, it is likely that persons who were too cognitively impaired or unable to comprehend English-language study materials were excluded. Patients received no compensation for participation.
Part I of the CAT-CPI is a fixed-length set of items measuring the frequency with which people participate in 20 activities, the importance of each activity, and the level of satisfaction with the activity frequency. Respondents saw only 1 item on each screen and could not skip questions. Part II consisted of 2 item banks measuring community enfranchisement.
The first bank measures the importance of participation, and the second bank assesses control over participation. We used the same CAT algorithm for both banks. We set the minimum number of items to be administered for each bank at 5. Patients were able to log in to the system repeatedly to complete the CAT-CPI. We merged data from the CAT-CPI with the hospital database that included the Inpatient Rehabilitation Facility Patient Assessment Instrument
We defined 2 outcome variables for this study. The first outcome was completion of the CAT-CPI, defined as logging into the system and answering all items; we defined an unsuccessful outcome as not logging into the system or failing to complete all items. The second outcome was the mode of Community Participation Indicators administration (Internet or IVR).
We selected variables that might affect completion of a CAT-administered PRO measure based on a review of the literature and our clinical experience.
We used 2 sets of predictor variables. First, demographic characteristics included patients' age, race, marital status, premorbid vocational status, sex, diagnosis, tier group, and primary payer. Second, rehabilitation variables included inpatient rehabilitation hospital LOS, overall satisfaction with care, discharge setting, and discharge motor and cognitive status. We used the hospital's patient satisfaction instrument to measure patient satisfaction.
Patients rated satisfaction items on a 4-point rating scale of excellent (4), good (3), fair (2), and poor (1). We used a single item (overall satisfaction with care) because an earlier study indicated that it correlated strongly with the total score and reflected patients' global perceptions.
We used counts, percentages, means, and SDs to describe patient characteristics; multivariate logistic regression analyses were used to examine associations between demographic, functional, and rehabilitation characteristics with the completion of the CAT-CPI and the mode of administration. We calculated odds ratios (ORs) and 95% confidence intervals (CIs) and specified statistical significance (α=.05). We used listwise deletion of cases with missing data. We examined group differences in time required for completing the 2 modes of CAT administration. We computed the number of days required for test completion by subtracting the date when all survey items were completed from the date we mailed the postcard; we compared groups computing Student t tests and Cohen d statistics. We also examined the characteristic of patients who did not log in to the system and those who dropped out. We completed all analyses with SPSS version 20.0a and Winsteps version 3.68.0.b
Figure 1 shows the flow diagram of patients. A total of 3221 patients met the eligible criteria (table 1). The 674 patients who were asked and agreed to complete the CAT-CPI represent 21% of the patients who were interviewed during the study period, without any incentive. Because outcomes management staff forwarded the contact information only for patients who agreed to complete the CAT-CPI, we cannot distinguish patients who refused from those who were not asked to complete the CAT-CPI. We estimate that <5% of eligible patients were not invited, primarily because of a part-time staff member who was hired after the study began. Therefore, it is important to evaluate the representativeness of the patients who agreed to complete the CAT-CPI.
We compared patients who agreed and those who refused or were not asked on 13 demographic, functional, and rehabilitation characteristics. Because of the large sample, relatively small differences are statistically significant (9 of 13 comparisons, P<.01). Therefore, we examined effect sizes. All comparisons yielded very small to small effects, except for 2 comparisons that were close to medium effects (Cohen d ≥|.35| and <|0.5|). Patients who refused/were not asked to complete the CAT-CPI were slightly older (69 vs 63y) and reported slightly lower satisfaction with overall care (mean, 3.6 vs 3.8 on a rating scale in which 3 means good satisfaction and 4 means excellent satisfaction) than patients who agreed to complete the CAT-CPI. The 674 (21%) eligible patients discharged from the hospital agreed to complete the CAT-administered CPI. The distribution of impairments shows that 34.4% had neurologic conditions, 43.2% had orthopedic conditions, and 22.4% had medical conditions. Of those agreeing to complete the CAT-CPI, 231 (34.3%) logged into the system and completed all items. One hundred forty-one (61%) participants chose IVR, whereas 90 (39%) chose Internet administration. Of the remaining 443 patients, 399 (90%) did not log in to the system and 44 (10%) stopped before completing all items. We compared the demographic, functional, or rehabilitation characteristics between these 2 subgroups (table 2). Among all comparisons, only age was statistically significant. Patients who stopped before completing all items were slightly older (68 vs 62y) than patients who did not log in to the system. Among the patients who did not finish the CAT, 40 (91%) chose IVR administration, whereas only 4 (9%) chose Internet administration.
Table 2Characteristics of patients who did not complete the CAT-CPI
We modeled patients' completion of the CAT-CPI using 13 demographic, functional, and rehabilitation variables to predict the likelihood of CAT-CPI completion.
The omnibus test for the logistic regression model was statistically significant [χ2(23)=63.398, p<0.001]. The Nagelkerke R2 was .14. The Hosmer-Lemeshow goodness-of-fit test was not statistically significant [χ2(8)=15.289, p=0.054]. This P value suggests that the model fit is marginal, indicating that the results may not be replicable. We entered demographic characteristics in the first step. Compared with white patients, black patients had a 57% reduced likelihood of completing the CAT-CPI (OR=.43; 95% CI, .25–.76); patients from other racial backgrounds had an 87% reduced likelihood (OR=.13; 95% CI, .04–.40). Compared with patients with spinal cord injuries, patients with stroke had a 55% reduced likelihood of completing the CAT-CPI (OR=.45; 95% CI, .21–.98). Patients with brain injury had an 81% reduced likelihood of completing the CAT-CPI (OR=.19; 95% CI, .07–.52). Patients with orthopedic conditions had a 49% reduced likelihood of completing the CAT-CPI (OR=.51; 95% CI, .27–.97). Patients with other impairments had a 64% reduced likelihood (OR=.36; 95% CI, .18–.73) of completing the CAT-CPI. Medicaid beneficiaries had a 74% reduced likelihood of completing the measure (OR=.26; 95% CI, .09–.72) compared with Medicare beneficiaries.
In the second step, we added 5 functional and rehabilitation variables as predictors. Black patients, patients from other racial backgrounds, patients with other impairment, and Medicaid beneficiaries remained less likely to complete the CAT-CPI; LOS and discharge FIM cognition measure were associated with a greater likelihood of completing the CAT-CPI (table 3). A 1-unit increase in rehabilitation LOS (OR=1.02; 95% CI, 1.001–1.05) was associated with 2% increased odds of completing the CAT-CPI. A 1-unit increase in discharge FIM cognition score (OR=1.07; 95% CI, 1.01–1.13) was associated with a 7% increased odds of completing the CAT-CPI. The wide range of 95% CIs around the ORs appeared in most of these significant relations, suggesting that these ORs lack precision; these relations should be interpreted cautiously.
Table 3Patient characteristics associated with completion of the CAT-CPI (n=614)
We modeled the selection of Internet versus IVR modality for patients who completed the CAT-CPI using the same set of predictors. The omnibus test for the logistic regression model was statistically significant (χ223,207=51.73, P=.001). The Nagelkerke R2 was .30, indicating that 30% of the variance in choosing Internet administration is explained by the predictors. The Hosmer-Lemeshow goodness-of-fit test was not statistically significant (χ28,207=4.133, P=.845), indicating that the model fits the data well. In the first step, older patients were 4% less likely (OR=.96; 95% CI, .93–.999) to choose Internet administration. Compared with patients who worked before illness, retired patients were 67% less likely (OR=.33; 95% CI, .12–.90) to choose Internet administration. Women were 54% less likely than men to choose Internet administration (OR=.46; 95% CI, .23–.93).
After we entered the functional and rehabilitation variables, retired patients remained less likely to select Internet administration. Race and discharge FIM motor measure were related to mode of administration (table 4). Black patients had a 77% lower likelihood of choosing Internet administration (OR=.23; 95% CI, .06–.92) than white patients. A 1-unit increase in discharge FIM motor scores (OR=1.04; 95% CI, 1.002–1.09) was associated with 4% increased odds of choosing Internet administration.
Table 4Patient characteristics associated with the choice of Internet administration (n=207)
Patients who chose Internet administration, since the first postcard was mailed, took longer to log into the system and complete the participation measure (11.9±9.8d) than patients who chose IVR (9.0±9.8d; t226,228=−2.157, P=.032). Cohen d was .29, indicating a small effect size. Although more patients chose IVR administration, a smaller proportion of this group completed the CAT (141 out of 181 patients or 78%) than patients who selected the Internet option (90 out of 94 patients or 96%; χ21,275=14.658, P<.001).
The purpose of this study was to examine the feasibility of a CAT-administered PRO for data collection after rehabilitation hospitalization and examine patient, functional, and rehabilitation characteristics associated with completion of the CAT-administered PRO and the mode of administration selected. We examined these questions prospectively in a cohort of patients who were discharged from inpatient rehabilitation. Twenty-one percent of the eligible patients agreed to complete the CAT-administered PRO without an incentive; among these patients, 34% completed the CAT-administered PRO. Therefore, only 7% of the eligible patients actually completed the CAT-administered PRO. Those agreeing and declining to complete the CAT-administered PRO were similar in most demographic, rehabilitation, and functional characteristics, except that older patients and those reporting slightly lower overall satisfaction were less likely to agree to complete the CAT-administered PRO. Although the study design does not address why patients declined, staff feedback suggests that feeling burdened by voluntary opportunities to provide postdischarge health outcomes was not a priority.
The low response rate suggests that the use of the CAT platform for collecting postrehabilitation PROs may not be a viable option for most rehabilitation populations without an incentive. Earlier studies
that found CAT-administered PROs are feasible in many clinical settings; however, the use of a CAT platform for collecting PROs after rehabilitation hospitalization may pose greater challenges, perhaps because of the heterogeneous nature of rehabilitation patient populations. Despite these challenges, our findings may support the feasibility of CAT-administered PROs for postrehabilitation data collection in a subset of patients, including those with higher cognitive function at discharge, longer LOSs, spinal cord injury, nonminority status, and nonpoverty level incomes. Focusing on patients with these characteristics allows for a more representative postdischarge rehabilitation sample. Further research is needed to support such a conclusion.
Prior rehabilitation studies regarding CAT applications have focused primarily on the evaluation of instruments' responsiveness to change and respondent burden among rehabilitation patients.
This investigation extends prior studies by examining patient demographic, rehabilitation, and functional characteristics associated with the completion of a CAT-administered PRO. Lower completion rates were associated with minority status, diagnosis other than spinal cord injury, Medicaid insurance, shorter LOSs, and lower cognitive function at discharge. Medicaid insurance is probably a proxy for low-income status. These associations underscore the risk of selection bias in the exclusive use of Internet and IVR for data collection. Although access to the telecommunication technologies has increased (eg, 77% of the U.S. population has Internet access),
Special effort is required to assure that lower income patients are able to access technology used in CAT-administered instruments. IVR is an important option given that more respondents in this study selected the IVR option and that telephone access is greater than Internet access.
Among those who completed the CAT-CPI, most (61%) selected and completed the IVR option; patients who completed the CAT-CPI using the IVR were 3 days quicker, on average, than those using the Internet. Preference for IVR was associated with older age, retirement status, being a woman, minority status, and lower motor function at discharge. Although more patients chose IVR administration, a smaller proportion of this group who started the CAT actually completed it (78%) compared with patients who selected the Internet option (96%). It may be that the IVR administration imposes greater challenges or is more burdensome because of the time required. Implementing routine PRO measurement involves a number of considerations, including selection of the appropriate patients, settings, timing of assessment, and the optimal mode of administration.
Offering both IVR and Internet modalities may help minimize biased sampling and increase patient participation. A qualitative or mixed-methods approach could help us understand and address the reasons for not completing the CAT.
Although it is feasible to use a CAT platform for collecting postrehabilitation PROs for a select group of patients, their responses may be biased because of age and satisfaction with care. Patients who agreed to complete the CAT-administered PRO reported higher satisfaction with overall care, highlighting that the rehabilitation care experience may affect patients' motivation to complete and provide feedback after discharge.
Generalizing the study conclusions is limited by several factors. Patients were recruited from 1 rehabilitation hospital in a metropolitan area; therefore, results may not generalize to persons living in other regions. This study protocol invited only the patients who provided self-reports on the telephone-based satisfaction survey; patients with severe cognitive and motor function, those with communication disabilities, and those without telephone service were excluded from this study. A major difficulty with self-administered PROs is the cognitive and communication skills required to answer items independently and in a reliable manner. Data collection methods that require cognitive and communication skills may disenfranchise individuals and create selection bias that limits the usefulness of study data.
Future studies should develop a multilanguage CAT platform for collecting postrehabilitation PROs to permit broader accessibility, particular for culturally diverse patient populations who speak little English.
Patients might have received assistance to complete the CAT-CPI, which may influence the results. Asking patients to complete the CAT-administered measure immediately after they completed the hospital's patient satisfaction instrument may create bias. Response compliance may be enhanced if CAT-administered PROs are administered in a routine postrehabilitation outcome measurement protocol. Clear guidelines are needed in the use of CATs. The low response rate limits generalization of the findings to the entire rehabilitation population. Suitable incentives may enhance patients' motivation for participation in research and clinical practice initiatives, thereby providing greater opportunities for outcome assessment after hospitalization. Results on patient characteristics associated with completion of the CAT-administered measure should be interpreted cautiously. Effective incentives are needed to increase the response rate and the representativeness of the sample. We did not document whether the nonrespondents (ie, those who did not log in to the system) would have chosen the same mode of administration again. Future studies should evaluate CAT-administered participation measures in routine, postrehabilitation quality monitoring protocols and examine which patient characteristics are associated with the selection and completion of different modes of CAT administration. Reasons for declining to complete CAT-administered PRO measures should also be studied.
This study supports the feasibility of CAT-administered PROs by Internet or IVR administration for collection of postrehabilitation PROs for a specific subset of patients. PRO completion was associated with cognitive function, LOS, payer, diagnosis, and minority status. Modality preference was associated with motor function, retirement status, and minority status. Providing alternative ways of answering questions helps assure that a larger proportion of patients have the ability to respond.
IBM North America, 590 Madison Ave, New York, NY 10022.
We thank Holly DeMark Neumann, MPPA, at the Rehabilitation Institute of Chicago for research coordination, and Chih-Hung Chang, PhD, at the Rehabilitation Institute of Chicago, for insightful comments on an earlier draft of this manuscript.
Ware Jr., J.E.
Item response theory and computerized adaptive testing: implications for outcomes measurement in rehabilitation.
Supported by the National Institute on Disability and Rehabilitation Research through a rehabilitation research training center on measuring rehabilitation outcomes and effectiveness grant (no. H133B040032 ) awarded to the Rehabilitation Institute of Chicago.
No commercial party having a direct financial interest in the results of the research supporting this article has conferred or will confer a benefit on the authors or on any organization with which the authors are associated.