Dimensions of Disordered Attention in Traumatic Brain Injury: Further Validation of the Moss Attention Rating Scale

Participants 

Participants were 372 inpatients with moderate to severe TBI, admitted for acute rehabilitation to 8 Traumatic Brain Injury Model System (TBIMS) centers over a 3-year period (May 1999−May 2002). Most of the participants (315, 85%) were enrolled in the TBIMS longitudinal database project supported by the National Institute on Disability and Rehabilitation Research.23 The remaining 57 participants, who were included from 3 of the collaborating centers, were not enrolled in the TBIMS database because they were treated acutely in a non-TBIMS trauma center before their admission to the TBIMS rehabilitation center. In other respects, they were comparable to the TBIMS sample, and met all of the following inclusion criteria. All participants had sustained a TBI as evidenced by loss of consciousness, focal brain lesion on neuroimaging, and/or abnormality on neurologic examination consistent with external trauma; all were at least 16 years of age. Further inclusion criteria stipulated a Rancho Los Amigos Level of Cognitive Functioning (RLA) score greater than III at the time of rating on the MARS, and stable dosing levels on all psychotropic medications (if any were prescribed) during the rating period. Because all participants were rated on the MARS by their treating occupational and physical therapists over a 3-day period (see below), patients who were otherwise eligible were excluded if they were not receiving both occupational therapy (OT) and physical therapy (PT) treatment, or if those treatments were given entirely in an unusual setting such as a private treatment room, or if there had been changes in their assigned OT or PT staff during the rating interval. We did not obtain separate informed consent for this study because it was subsumed under the overall TBIMS informed consent process. The 57 participants not included in the TBIMS were rated anonymously under an informed consent waiver on the condition that no information be recorded about them other than their MARS scores. Therefore, descriptive information about the sample as a whole could be derived only from the 315 TBIMS enrollees.

The demographic and injury characteristics of the TBIMS study sample are summarized in table 1. Not all injury severity indices were available for all participants. The Glasgow Coma Scale (GCS)24 was scored on admission to the emergency department in the acute care hospital. Time to follow commands was defined as the interval in days between the TBI and the date the patient followed simple commands on 2 consecutive assessments within 24 hours. As is consistent with other samples of moderate to severe TBI, the majority of participants were male (70%), and most had been injured in motor vehicle collisions (also 70%). Admission GCS scores varied across the range of the scale, although the average score (8) indicated severe injury, as did the average time to follow commands greater than 1 week. Because the sample consisted of patients admitted for inpatient rehabilitation, cases with high GCS scores were likely to have had complications such as subdural hematoma, or to have sustained focal injury such as a gunshot wound to the brain. Participants were scored on the MARS from 8 to 99 days postinjury (mean, 33d).

Table 1. Characteristics of the TBIMS Sample (n=315)

	Characteristics	Values
	Age at injury (y)
	Mean ± SD	37.0±18.3
	Range	16–90
	Sex, n (%)
	Male	222(70)
	Female	93(30)
	Race, n (%)
	White	222(70)
	African American	73(23)
	Other ethnic minority	20(7)
	Etiology
	Motor vehicle collision (includes pedestrian injuries), n (%)	222(70)
	Fall	48(15)
	Blunt assault	21(7)
	Gunshot wound	8(3)
	Sports/other	14(4)
	Missing/unknown	2(<1)
	GCS on emergency admission (n=265)
	Mean ± SD	8.7±4.2
	Median	8.0
	Range	3–15
	Time to follow commands (d) (n=301)
	Mean ± SD	8.4±11.6
	Range	0.5–76.0

Abbreviations: GCS, Glasgow Coma Scale; SD, standard deviation.

Instrument 

The research version of the MARS is a 45-item, Likert-type rating scale; its development process has been previously described.17 (The full research version of the scale included 8 “control” or filler items not directly related to attention, for a total of 53 items. The filler items, however, were not analyzed in this study.) The response choices lie on a 5-point scale, ranging from “definitely true” to “definitely false” with an intermediate rating of “sometimes true, sometimes false.” Some items are written so that the “false” response choices indicate an attention deficit (eg, maintains eye gaze appropriate to task at hand), while for other items, “true” responses indicate an attention deficit (eg, persists with an activity or response after being told to stop).

Procedure 

The study protocol called for patients to be rated on the MARS over a 3-day period independently by their treating occupational and physical therapists, with ratings to be completed between day 3 and day 17 after admission to acute rehabilitation. Thus, the last day for the rating period to begin would be day 15. We chose this arbitrary time window to ensure that the rating therapists would be familiar with the patient and to minimize the possibility that they might have known a given patient across widely differing functional levels, which might contaminate ratings. As the project progressed, the upper end of the window was relaxed somewhat to include severely impaired participants, some of whom became more responsive (and thus met the RLA score > III criterion) later than 15 days into their rehabilitation stay. In a few other cases, participants were rated beyond the 17-day cutoff because of experimenter error in calculating the interval, or because of administrative or logistical problems. Of the 315 participants for whom data were available with which to calculate the admission-to-rating interval, 279 (89%) were rated within the 3- to 17-day window. Four participants were inadvertently rated on days 1 and 2, 23 were rated between days 18 and 21, and the remaining 7 were rated beyond that time (days 22–71). We decided to retain all early and late ratings in the analysis because of the small number of outliers.

On enrollment of eligible patients into the TBIMS (or admission to acute rehabilitation, for the participants who were not in the TBIMS), a research coordinator attempted to identify a rating interval of 3 consecutive days within the required time window during which the OT and PT staff would be constant, and during which psychotropic medications would also be stable. In some institutions this meant meeting in advance with the physician to discuss medication plans; in other centers, the ratings were carried out regardless but discarded if medication changes were documented during the rating interval. The research coordinator in each institution met with the physical and occupational therapists to explain the project and answer questions about the rating instrument. Beyond repeating this information as necessary, no formal training was conducted on how to use the MARS. Each therapist was asked to complete the rating form independently, based on his/her interactions with and observations of the patient during the course of routinely scheduled therapy sessions during the 3 days. Deidentified rating sheets were faxed to the lead center, where the data were stored in an Access database.a

Data Analysis 

Descriptive Findings 

Summary statistics on the therapist MARS ratings are displayed in table 2. The distributions of ratings in the 2 disciplines were similar (ie, means were approximately equal to medians and standard deviations were comparable). As reported previously from an analysis of a smaller sample,17 physical therapists rated patients slightly, but systematically higher (more attentive), than occupational therapists. This difference was statistically significant using a paired t test (t=−2.8, P<.01). The OT and PT ratings correlated at r equal to .68 (P<.01).

Table 2. Occupational and Physical Therapists’ MARS Ratings for the Entire Sample (N=372)

	MARS Scores	Occupational Therapists	Physical Therapists
	Mean	157.5	162.9
	Median	159.0	165.5
	Range	75.0–219.0	82.0–222.0
	SD	30.2	29.5

NOTE. Possible scores range from 45 to 225 with higher scores indicating better attention.

Exploratory Factor Analysis 

Results of the first-pass EFA on the PT sample suggested that 4 factors should be extracted. The fourth factor, however, had only a few items, and those also had significant cross-loadings on other factors. Hence, we retained 3 factors and used an oblique promax rotation to achieve simple structure. We retained items that tended to load clearly on a single factor as long as the groupings made conceptual or theoretical sense. The 3 factors were interpreted as follows: factor 1 contained items describing restlessness/distractibility (10 items); factor 2 was termed initiation (4 items); and factor 3 seemed to represent sustained/consistent attention (5 items).

Confirmatory Factor Analysis 

We evaluated the 19-item scale derived from the EFA and its associated 3-factor model, using the OT sample via CFA. The data had substantial non-normality, even after we attempted to normalize the items prior to our analysis. Thus, we derived a polychoric correlation matrix and an asymptotic covariance matrix, and used the weighted least squares (asymptotic distribution free) method to estimate the parameters in all 3 CFA models. In our primary 3-factor model, we allowed the error terms to correlate only among items loading on the same factor. Correlated error terms represent the hypothesis that unique variances of the associated indicators overlap. That is, they measure something in common other than the latent constructs that are represented in the model. In this case, it is likely that items loading on the same factor may be susceptible to the same response set.

Table 3 presents a panel of fit indexes for all 3 models, based on recommendations by Hu and Bentler29 for assessing model fit derived from stimulation. Both global and component fit indexes indicated that our primary 3-factor model was the best fitting model. For example, the non-normed fit index, comparative fit index, and relative fit index all exceeded .95, RMSEA was less than .06, and standardized root mean squared residual was less than .08. The Bayesian information criterion difference of 290.83 between the 3-factor model and the next best fitting model—the 1-factor model—provided strong support for preferring the 3-factor model over the alternative models.30 The second-order model was particularly problematic in terms of Heywood cases, that is, negative variance estimates. Attempts to provide alternative start values prior to model estimation did not remedy this situation. Hence, we restricted our further focus to the 3-factor model.

Table 3. Summary of Fit Indexes for Measurement Models

	Model	χ2 Test (df)	P	NNFI	CFI	RNI	BIC	AIC	RMSEA	90% CI RMSEA	SRMR
	3-factor	462.83(134)	.00	.98	.99	.97	794.29	574.83	.05	.04–.06	.06
	1-factor	771.42(137)	.00	.95	.97	.94	1085.12	844.42	.08	.07–.09	.08
	Second-order	1078.92(165)	.00	.94	.95	.93	1226.89	1128.92	.09	.08–.10	.09

Abbreviations: AIC, Akaike information criterion; BIC, Bayesian information criterion; CFI, comparative fit index; CI, confidence interval; NNFI, non-normed fit index (Tucker-Lewis coefficient); RNI, relative fit index; SRMR, standardized root mean squared residual.

Table 4 shows the structure coefficients and squared multiple correlations coefficients for the items. The correlation coefficient can be viewed as a measure of reliability whereas the structure coefficients assess aspects of validity in determining which items are strong indicators of the latent construct. The numerals in bold indicate which items loaded on which factor. For example, 4 items loaded on the sustained/consistent attention factor and not on the other 2 factors. It should be pointed out that constraining an item to load on only 1 factor does not constrain the factor structure coefficients to be zero if the factors are correlated. In fact, examining the factor structure coefficients associated with all of the factors can be essential for accurate interpretation.

Table 4. Structure Coefficients and Squared Multiple Correlations for 3-Factor Model

	Item	RD	Initiation	SC	R2
	Is restless or fidgety when unoccupied	.66	(.32)	(.51)	.44
	Tends to speak less than he/she is capable of	(.19)	.39	(.27)	.15
	Performance is best early in the day or after a rest	(.15)	(.13)	.19	.03
	Initiates communication with others	(−.31)	−.63	(−.43)	.40
	Is restless or fidgety during tasks	.71	(.35)	(.55)	.51
	Persists with an activity or response after being told to stop	.73	(.36)	(.56)	.53
	Interrupts tasks to express physical complaints that are unrelated to task	.62	(.30)	(.48)	.38
	Attends to nearby conversations rather than the current task or conversation	.63	(.31)	(.49)	.40
	Tends not to initiate tasks which are within his/her capabilities	(.43)	.87	(.59)	.75
	Speed or accuracy deteriorates over several minutes on a task, but improves after a break	(.34)	(.30)	.44	.19
	Is able to perform or interact for only a few moments before fatiguing or stopping	(.60)	(.53)	.78	.61
	Makes unnecessary comments/conversation that interferes with task performance	.75	(.37)	(.58)	.57
	Performance of comparable activities is inconsistent from one day to the next	(.49)	(.44)	.64	.41
	Perseverates on previous topics of conversation or previous actions	.77	(.38)	(.59)	.59
	Initiates activity (whether appropriate or not) without cueing	(−.44)	−.89	(−.61)	.80
	Sustains activity even when people enter the room	−.74	(−.36)	(−.57)	.55
	Begins to touch or manipulate nearby objects not related to task	.75	(.37)	(.58)	.56
	Performance of comparable activities is relatively consistent from 1 time of day to another	(−.59)	(−.52)	−.77	.60
	Tends to focus on details that are irrelevant for the task or activity	.81	(.40)	(.62)	.65

NOTE. Values in parentheses are the structure coefficients for the factors whose pattern coefficients were constrained to equal zero. Values in boldface indicate which items loaded on which factor. Abbreviations: RD, restlessness/distractibility; SC, sustained/consistent attention.

Analysis of the parameter estimates for the 3-factor model revealed no inadmissible estimates (eg, correlations exceeding 1.0) or Heywood cases (ie, negative covariances). All parameter estimates were statistically significant. Note, however, the R2 values for 2 items on the sustained/consistent attention factor (Performance is best early in the day or after a rest, Speed or accuracy deteriorates over several minutes on a task, but improves after a break) and 1 item on the initiation factor (Tends to speak less than he/she is capable of) were low (<.20). This suggested unreliability or a lack of commonality with the other items on their respective factors. The sustained/consistent factor correlated highly with the restlessness/distractibility factor (r=.75) and the initiation factor (r=.66). Initiation and restlessness/distractibility correlated moderately (r=.46). In addition, examination of the factor structure revealed that most MARS items had substantial shared variance across factors; for example, item one (Performance of comparable activities is relatively consistent from 1 time of day to another), coefficients were −.59, −.52, and −.77.

In summary, the combined results of the EFA and CFA suggested evidence for the hypothesized multidimensionality in the MARS. The best fitting model, however, produced 3 factors with relatively high intercorrelations among factors, and shared item variance across factors. From a psychometric standpoint, the restlessness/distractibility factor was the strongest factor, with a larger number of coherent items than the other 2 factors.

Rasch Analysis 

As previously noted, one purpose of our study was to reduce redundancy in item content and minimize rater burden, but to still produce a reliable estimate of attention and adequately represent the identified attention subdomains. An item set that could achieve these goals would need to cover a wide range of difficulty (ie, without floor or ceiling effects) with items spaced throughout the distribution, avoiding gaps. A gap is defined as a difference in adjacent item difficulty estimates that is greater than their joint standard errors of measurement.

The Rasch analyses were begun with the items that measured each of the 3 identified attention factors. This set of 19 items exhibited good separation and reliability (person, 2.33/.84; item, 4.97/.96; using a criterion of >2.00) and good overall fit, although 1 item (Speaks as much as is capable of) significantly misfit (infit, 1.95; outfit, 2.02; using a criterion of >1.40). We then undertook to delete factor items that were redundant with regard to difficulty level, and to add items from the original MARS set that filled gaps in the overall coverage of the 19-item scale. Items were selected from each factor that covered an adequate range of content and difficulty, and were recombined into a single scale for reanalysis. To address shortcomings in the overall distribution of item difficulty, items that did not load on 1 of the 3 factors but fit the overall attention domain were identified and added to the item set. These additional items were selected both for their difficulty level, their loading patterns on more than 1 identified factor, and their face validity for including item content that was not already represented by factor items. For example, some “higher-level” items, such as detecting one’s own errors, had shown multifactor loadings and thus were not considered members of any of the 3 factors, yet were considered important aspects of attention to be included.

Altogether in these calibrations, 5 items were deleted from the original 3 factors (3 redundant items from factor 1, 1 misfitting and redundant item from factor 2, 1 redundant item from factor 3); 4 nonfactor items were added to increase coverage at the extremes; 6 nonfactor items were added to fill gaps in the difficulty distribution across the rating scale; and 2 items that misfit after these additions and deletions were deleted. When these changes were made, a set of 22 items was identified that contained no item misfit or distributional gaps. Eleven of these represented the 3 factors, with the other 11 items drawn from the original MARS. The items and their associated factors are shown in table 5. These 22 items were able to differentiate people into at least 3 distinct strata of attention (person separation and reliability, 2.60/.87) and covered a wide range of content and item difficulty (item separation and reliability, 8.04/.98). Because they contained so few items, the factors were not expected to reliably differentiate people along the attention scale; however, despite their brevity, these factors covered an acceptable range of difficulty. Person and item separation and reliability for the 3 factors was 1.31 and 0.63 and 3.42 and 0.92 for restlessness/distractibility; 1.23 and 0.60 and 3.42 and 0.92 for initiation; and 0.89 and 0.44 and 7.06 and 0.98 for sustained/consistent attention. (More detailed information about the results of all data analyses, including full results of the EFA, Rasch variable maps, and items not retained in the scale, are available from the first author on request.)

Table 5. Items of the Revised MARS and Their Associated Factors

	Item	Factor
	Is restless or fidgety when unoccupied	RD
	Sustains conversation without interjecting irrelevant or off-topic comments	NA
	Persists at a task or conversation for several minutes without stopping or “drifting off”	NA
	Stops performing a task when given something else to do or to think about	NA
	Misses materials needed for tasks even though they are within sight and reach	NA
	Performance is best early in the day or after a rest	SC
	Initiates communication with others	IN
	Fails to return to a task after an interruption unless prompted to do so	NA
	Looks toward people approaching	NA
	Persists with an activity or response after being told to stop	RD
	Has no difficulty stopping one task or step in order to begin the next one	NA
	Attends to nearby conversations rather than the current task or conversation	RD
	Tends not to initiate tasks which are within his/her capabilities	IN
	Speed or accuracy deteriorates over several minutes on a task, but improves after a break	SC
	Performance of comparable activities is inconsistent from 1 day to the next	SC
	Fails to notice situations affecting current performance, eg, wheelchair hitting against table	NA
	Perseverates on previous topics of conversation or previous actions	RD
	Detects errors in his/ her own performance	NA
	Initiates activity (whether appropriate or not) without cueing	IN
	Reacts to objects being directed toward him/her	NA
	Performs better on tasks when directions are given slowly	NA
	Begins to touch or manipulate nearby objects not related to task	RD

NOTE. Items are listed in the order in which they appeared in the 45-item research version. Abbreviations: IN, initiation; NA, not applicable (retained in the MARS, but not included in any of the factor subscales); RD, restlessness/distractibility; SC, sustained/consistent attention.

The 22-Item Version of the MARS 

In summary, results of data analysis yielded a new MARS that is less than half the length of the research version and provides: full coverage of the attention dimension represented by the original scale and identified by Rasch analysis; 3 subscores for the correlated factors identified by the EFA and CFA; and a total score that incorporates all 3 factors, plus additional items that round out the difficulty level and multidimensionality of the construct measurement.