Assessing Self-Care and Social Function Using a Computer Adaptive Testing Version of the Pediatric Evaluation of Disability Inventory

Analytic sample 

We used an existing database of 881 children who had complete data on the 73-item self-care and the 65-item social function scales of functional skills part of the PEDI. This retrospective analytic sample included 2 groups: (1) a normative sample of 412 healthy children between the ages of 6 months and 7.5 years that was also used to create the initial standardization and normative scoring of the PEDI, and (2) a clinical sample of 469 children and youth (age range, 6mo–17y) who had received inpatient, outpatient, or school-based rehabilitation services at Franciscan Hospital for Children, Boston, MA. Of the 469 clinical cases, 249 had longitudinal data appropriate for sensitivity analyses for the self-care scale and 200 had data for the social function scale.

Approximately 48% of the children in the clinical sample had congenital or inherited diseases, 21% had growth and maturation disorders, 16% had acquired conditions, and 15% were diagnosed with traumatic injuries. Demographic characteristics of the analytic sample are presented in table 1. The sample size of 881 is acceptable for initial calibration work for a prototype CAT.12

Table 1. Demographic Characteristics of Samples

	Characteristics	Analytic Sample	Cross-Validation Sample
	Age range	6mo–17y	6mo–18y
	% Female	45.2	49.3
	% Hispanic or Latino	9.3	5.5
	% Asian	1.5	5.5
	% Other	5.8	4.1
	% Black or African American	14.6	2.7
	% White	68.8	82.2
	Total sample size	881.0	73.0

Cross-validation sample 

We recruited a convenience sample of 73 children and youth for the prospective cross-validation study. Thirty-eight children with disabilities, ages 1 year to 17 years, were recruited from the clinical programs (inpatient, outpatient, early intervention, and hospital-based school) at Franciscan Hospital for Children. Ethnic representation corresponding to the current United States census was targeted for recruitment; however, respondents who did not speak English as a primary language were excluded because of the prohibitive cost of translating and interpreting. Children were further selectively recruited to assure representation of each of the following 4 impairment groups: congenital or inherited disease, growth and maturation disorders, acquired conditions, and traumatic injuries. Thirty-five children without disabilities, ages 6 months to 7.5 years, were recruited through the Franciscan Family Child Care Center and the home communities of the 2 field-test coordinators.

Unidimensionality and local independence 

IRT and CAT methods assume certain measurement properties of item sets that purport to represent a functional construct (latent variable). These include the assumptions of unidimensionality, local independence, and stability of item parameters across groups (eg, clinical vs normative samples). Item sets that violate these assumptions may be less effective in modeling the latent variable and may limit the accuracy of a CAT instrument. A key assumption of the latent variable models that serve as the basis for CAT is that all items in a scale measure a single, unitary concept; that is, the items are unidimensional. The latent variable alone should explain how items are related to one another.32, 33 We tested the latent structure of the self-care and the social function items in a series of confirmatory factor analyses34 and evaluated item loadings and residual correlations between items using MPlus software.35, a We used weighted least squares means and variance adjusted estimation methods, which are more precise when analyzing moderate-size samples with skewed categorical data.34, 36 To determine the extent to which a unidimensional model adequately represented scale structure, we considered the eigenvalues associated with each factor extracted; item loadings on the primary factor; and results from overall model fit tests. To ensure adequate sample size for estimation of model parameters we combined the normative and clinical PEDI samples. Assuming the item parameters are similar across groups, combining the samples enhances generalizability of results across both groups and provides a greater number of persons at the moderate to low end of the scale to enhance precision of estimated scores in this region.

In the self-care domain, 1 factor explained 87.9% of the item variance and all the factor loadings were very high (range, .778–.974). The comparative fit index (CFI) value of .995 indicated very good fit and can be interpreted as an indicator that 99% of covariance in the data is reproducible by the model. This conclusion was supported by the Tucker-Lewis index (TLI) value of .997, also indicating good fit. The root mean square error of approximation (RMSEA) of .078 is in the acceptable range. In the social function domain 1 factor explained 87.8% of the item variance. All the factor loadings were very high, ranging from .77 to .987 and the fit indexes also supported the 1-factor model (CFI=.994, TLI=.997, RMSEA=.104).

The requirement of local independence means that scale items must be independent, or unrelated, to each other at a given score level. One indicator that items share more than the latent trait is high residual correlations. High residual correlations (> ±0.2) were observed between 9 pairs of items on the self-care scale and 24 pairs on the social function scale.37 These correlations likely reflect the structure of the PEDI, which groups similar items into skill sets that have an implicit hierarchical relation to each other. For example, the item “eats all textures of table food” implies accomplishment of the previous item “eats cut up/chunky/diced foods” and thus the response to the more challenging item is not independent of the response to the easier item. This violation of model assumptions may affect the estimation of test information and item discrimination parameters, but cannot be rectified in an existing database.

Item calibrations 

The item parameters for each scale were estimated using the Rasch model, which estimates the item difficulty parameters.38, 39, 40 The Rasch model was selected as the best solution for this phase of the project because of simplicity in interpretation and flexibility about the underlying form of the population or trait distributions. The item parameters and fit statistics were calculated using ConQuest,41, b which is based on marginal maximum likelihood estimation. We evaluated fit using the fit statistics for each item based on the comparison of expected and observed value. To maximize sample size and the distribution of item difficulty, data for the total analytic sample were used to generate item calibrations. Note that the original item calibration and instrument standardization for the PEDI was conducted using the normative sample alone (n=412).11

In the self-care domain there were 4 items that did not fit the model: “allows nose to be wiped” (infit=1.52), “removes socks and unfastened shoes” (infit=1.6), “manages tangles and parts hair” (infit=1.72), and “brushes or combs hair” (infit=1.68). Those items were removed from the item set to be used for the CAT prototype. In the social function domain only 1 item did not fit the model: “if upset because of a problem, child must be helped immediately or behavior deteriorates” (infit=1.81). Because of the important content reflected in this item we chose to keep it in the item pool. We estimated the individual scores using weighted maximum likelihood42 estimation. Weighted maximum likelihood is preferable to the expected a posteriori methods because it adjusts the first-order bias. The individual scores were standardized to a mean 50 and standard deviation (SD) of 10.

Differential item functioning 

In IRT, the child’s score on an item should depend entirely on the latent variable being measured. Significant differential item function (DIF) indicates that variables other than the latent variable, such as diagnosis, age, or sex, are likely influencing the response.43 We used logistic regression to determine the extent to which item responses to the self-care and social function items differed by clinical diagnosis or age. The diagnosis variable was treated dichotomously (clinical, typical) and age was treated as a continuous variable. If diagnosis or age produced significant model coefficients and the child variable explained more than 2% of variance, considering the total score, then an item was considered to exhibit DIF. A Bonferroni-corrected P value was applied for significance testing (self-care domain, P<.05/73=.0685; social function domain, P<.05/65 items=.0077). We also assessed the amount of model variance explained by the group variables.

One of the 73 self-care items (“removes socks and unfastened shoes”) exhibited DIF by diagnosis. This item also showed misfit on the previous analyses, thus supporting the decision to remove this item. Sixteen of the 65 social function items exhibited DIF by diagnosis or age. There were 2 items that functioned differently for both diagnosis and age: “if upset because of a problem, child must be helped immediately or behavior deteriorates” and “explores and functions in familiar community settings without supervision.” Because the problematic items represent important content, we did not remove them. However, these items are clearly candidates for future revision.

Development of the CAT program 

We based the self-care and social function CAT algorithms on the HDRI softwarec developed at the Health & Disability Research Institute. The CATs were designed to be completed by a child’s clinician or parent and can be administered from a stand-alone computer. We programmed the CATs to use weighted maximum likelihood score estimation.7 We selected the items “puts on pants with an elastic waist” and “provides names and descriptive information about family members” to be the first items administered to all respondents for the self-care and social function CATS, respectively. These items were chosen because their difficulty parameters were in the middle of the range, they did not exhibit DIF, and the content seemed appropriate for most respondents. The response to the first item is fed into the engine and the application calculates a probable score as well as a person-specific measure of how precise that score is. If the score is not estimated with sufficient precision, according to internal guidelines, additional questions are selected and administered until either the precision standard is reached or the defined maximum number of items has been administered. To be able to compare results from the simulation and cross-validation studies we used a fixed-stopping rule of 15 items in the present project. However, we expected that only a few respondents would need to complete that many items to attain desirable levels of precision.

Computer simulations 

We evaluated the IRT-based algorithms for each CAT using computer simulation methods for the analytic sample. The simulations compare the psychometric merits of alternative strategies for programming assessments. In these simulations, responses to items selected by the CAT software were obtained for cases in the analytic data set and fed to the computer to simulate the conditions of an actual CAT assessment. As in an actual CAT, the simulation uses the IRT model to select the best item to administer next, for example, the one with the highest information function given the current score level, re-estimates the domain score and CI, and decides whether or not to continue testing. In the present study, in order to be able to compare results from the simulation and cross-validation studies, we used a fixed-stopping rule of 15 items. We developed 3 CAT scores in the simulations to reflect 3 potential item-stopping rules (self-care or social function CAT-15, self-care or social function CAT-10, and self-care or social function CAT-5). These simulated scores were compared with a criterion standard—the actual IRT latent trait score (self-care or social function) estimated by the full model.

Cross-validation field test 

The self-care or social function CATs and full-length scales for each domain were completed on a sample of children with disabilities from the Franciscan Hospital for Children clinical programs through parent interview conducted by the field test coordinators. For children without disabilities, we also administered both instruments through interview with the parent or the parent’s designee (in some cases the child’s teacher or day care worker). The CAT was completed using the preset 15-item stopping rule to enable comparison with scores from the full-length scale. For all children, both the CAT and full-length scale were completed during 1 session. For both groups (children with and without disabilities), the order of assessment type was counterbalanced to avoid an order effect. After administration, we obtained verbal feedback from the physical therapist and/or parent respondent about the relative merits or limitations of both modes of administration. We collected the actual time (to the closest minute) required for administration of the full-length scale in 73% of the cases; each CAT had an internal clock to track the amount of time and the number of items needed to meet preset levels of precision. Demographic information (ethnicity, sex, age, and diagnosis when applicable) was collected for each child. All procedures were approved by the institutional review boards at Boston University and Franciscan Hospital for Children.