Advertisement

Predicting Fitness to Drive in People With Cognitive Impairments by Using DriveSafe and DriveAware

      Abstract

      Kay LG, Bundy AC, Clemson LM. Predicting fitness to drive in people with cognitive impairments by using DriveSafe and DriveAware.

      Objectives

      To examine the psychometric properties of DriveSafe and DriveAware and their predictive validity.

      Design

      Prospective study compared screening tests with criterion standard.

      Setting

      Two driving rehabilitation centers affiliated with a university and a geriatric rehabilitation facility.

      Participants

      Consecutive sample of drivers with functional impairments (n=115) and subgroup of drivers with cognitive impairments (n=96) referred for a driving assessment.

      Interventions

      Not applicable.

      Main Outcome Measure

      Driving performance was measured by a standardized assessment in real traffic.

      Results

      Rasch analysis provided evidence for construct validity and internal reliability of both tests. Tests trichotomized drivers into unsafe, safe, and further testing categories. The optimal lower cutoff identified unsafe drivers with a specificity of 97% (95% confidence interval [CI], 83–100) in the test sample and 96% (95% CI, 80–100) in the validation sample. The optimal upper cutoff identified safe drivers with a sensitivity of 93% (95% CI, 77–99) and 95% (95% CI, 76–100), respectively.

      Conclusions

      By using DriveSafe and DriveAware, drivers with cognitive impairments referred for a driving assessment can be categorized as unsafe, safe, or requiring further testing, with only 50% needing an on-road assessment. Before clinical practice is changed, these findings should be replicated.

      Key Words

      List of Abbreviations:

      CI (confidence interval), DIF (differential item function), MnSq (mean square), VRST-USyd (Visual Recognition Slide Test)
      DRIVING IS CONSIDERED TO be an essential activity of daily living with both functional and symbolic importance for people of all ages. For young people, it is a rite of passage enabling them to be independent. Similarly, for older people, driving is a source of identity and pride, symbolizing active participation in activities that provide meaning and purpose in life. However, at any stage in life, an accident or medical condition may cause deficits in any of the requisite skills that impact on a person's ability to drive a car.
      Currently, in many jurisdictions around the world, fitness to drive after the onset of a medical condition or accident is determined in a specialized driving rehabilitation center in which experienced professionals conduct extensive clinical and driving assessments either in real traffic or in a simulator over a period of 2 to 3 hours.
      • Mazer B.
      • Gelinas I.
      • Benoit D.
      Evaluating and retraining driving performance in clients with disabilities.
      The associated clinical evaluation commonly includes an interview to elicit information about driving history and requirements and medical history; an assessment of vision including acuity, visual fields, and eye movements; an assessment of physical capabilities and cognitive abilities; and knowledge of current road rules within the jurisdiction. In this context, a range of neuropsychologic tests also is used. Some centers refer all clients for a full neuropsychologic assessment,
      • Alvarez F.J.
      • Fierro I.
      Older drivers, medical condition, medical impairment and crash risk.
      • Ponsford A.S.
      • Viitanen M.
      • Lundberg C.
      • Johansson K.
      Assessment of driving after stroke—a pluridisciplinary task.
      whereas others use a small number of tests known to be specifically associated with safe or unsafe driving.
      • Akinwuntan A.E.
      • Feys H.
      • De Weerdt W.
      • Baten G.
      • Arno P.
      • Kiekens C.
      Prediction of driving after stroke: a prospective study.
      • Freund B.
      • Gravenstein S.
      • Ferris R.
      • Burke B.L.
      • Shaheen E.
      Drawing clocks and driving cars.
      Although comprehensive driving assessments provide clear answers about fitness to drive, they are almost universally conducted at the client's expense and with some degree of risk for those involved. Additionally, often the demand for these assessments is greater than the services can deliver resulting in long waiting lists. Consequently, there is a need to develop screening tests
      • Owsley C.
      Clinical and research issues on older drivers: future directions.
      that accurately trichotomize drivers into those who are unsafe or safe and those who require further testing.
      • Molnar F.J.
      • Patel A.
      • Marshall S.C.
      • Man-Son-Hing M.
      • Wilson K.G.
      Clinical utility of office-based cognitive predictors of fitness to drive in persons with dementia: a systematic review.
      Unsafe drivers can be counseled to cease driving; safe drivers can continue driving with regular reviews. The scarce resources of the driving rehabilitation centers can be used to provide assessment and rehabilitation for those who fall in the uncertain category.
      • Innes C.R.
      • Jones R.D.
      • Dalrymple-Alford J.C.
      • et al.
      Sensory-motor and cognitive tests predict driving ability of persons with brain disorders.
      This would represent a substantial cost saving for both the centers and drivers alike.
      Many of the neuropsychologic tests commonly used in driving assessments are statistically associated with safe or unsafe driving, but, to be effective as a screening tool, the tests also must be able to predict driving performance with acceptable accuracy.
      • Molnar F.J.
      • Patel A.
      • Marshall S.C.
      • Man-Son-Hing M.
      • Wilson K.G.
      Clinical utility of office-based cognitive predictors of fitness to drive in persons with dementia: a systematic review.
      To determine accuracy, the descriptive statistics of sensitivity, specificity, and positive and negative predictive values need to be calculated and reported to take into account errors in prediction.
      • Portney L.G.
      • Watkins M.P.
      Foundations of clinical research: applications to practice.
      In a systematic review of tests commonly used in driving assessment with clients with dementia, Molnar et al
      • Molnar F.J.
      • Patel A.
      • Marshall S.C.
      • Man-Son-Hing M.
      • Wilson K.G.
      Clinical utility of office-based cognitive predictors of fitness to drive in persons with dementia: a systematic review.
      found that none of the studies reviewed reported specific cutoff scores that generated these descriptive statistics. However, sensitivity and specificity are available for a small number of tests in studies with drivers with other impairments (table 1). Examination of table 1 reveals that few tests have adequate levels of both sensitivity and specificity. Some tests such as the Useful Field of View
      • Myers R.S.
      • Ball K.K.
      • Kalina T.D.
      • Roth D.L.
      • Goode K.T.
      Relation of useful field of view and other screening tests to on-road driving performance.
      that accurately identify unsafe drivers also falsely identify many capable drivers. Conversely, those tests that accurately identify safe drivers, such as Cognitive Behavioural Driving Inventory,
      • Bouillon L.
      • Mazer B.
      • Gelinas I.
      Validity of the Cognitive Behavior Driver's Inventory in predicting driving outcome.
      Clock Drawing Test,
      • Freund B.
      • Gravenstein S.
      • Ferris R.
      • Burke B.L.
      • Shaheen E.
      Drawing clocks and driving cars.
      and Maze Test,
      • Snellgrove C.A.
      Cognitive screening for the safe driving competence of older people with mild cognitive impairment or early dementia.
      also classify a high number of incapable drivers as safe. Other tests, including Trail Making Test part B
      • Reitan R.M.
      Trail Making Test: manual for administration, scoring and interpretation.
      • Mazer B.L.
      • Korner-Bitensky N.
      • Sofer S.
      Predicting ability to drive after stroke.
      Austroads
      Model Licence Re-Assessment Procedure for Older Drivers: Stage 2 Research.
      and Motor-free Visual Perception Test
      • Calarusso R.P.
      • Hammill D.D.
      Motor-Free Visual Perception Test.
      • Korner-Bitensky N.A.
      • Mazer B.L.
      • Sofer S.
      • et al.
      Visual testing for readiness to drive after stroke: a multicenter study.
      do not accurately discriminate safe and unsafe drivers. Tests designed for 1 diagnostic group (eg, Stroke Drivers Screening Test
      • Lundberg C.
      • Caneman G.
      • Samuelsson S.M.
      • Hakamies-Blomqvist L.
      • Almkvist O.
      The assessment of fitness to drive after a stroke: the Nordic Stroke Driver Screening Assessment.
      ) may not be applicable to others. Furthermore, some tests with promising results in an initial study (eg, Battery of 12 tests,
      • Wood J.M.
      • Anstey K.J.
      • Kerr G.K.
      • Lacherez P.F.
      • Lord S.
      A multidomain approach for predicting older driver safety under in-traffic road conditions.
      P-Drive,
      • Patomella A.-H.
      • Kottorp A.
      An evaluation of driving ability in a simulator: a good predictor of driving ability after a stroke?.
      SMCTest
      • Innes C.R.
      • Jones R.D.
      • Dalrymple-Alford J.C.
      • et al.
      Sensory-motor and cognitive tests predict driving ability of persons with brain disorders.
      ) have yet to be replicated. All of the tests were validated against an on-road assessment except the Clock Drawing Test, which was validated against simulator performance. Thus, none of the available tests is sufficiently accurate to be used as a screening test.
      Table 1Predictive Power of Off-Road Screening Tests
      Clinical TestAuthor, DateSample SizeDiagnosis of SampleSensitivity Fail Test With Fail Drive (%)Specificity Pass Test With Pass Drive (%)
      Cognitive Behavioural Driving InventoryBouillon et al, 2006
      • Bouillon L.
      • Mazer B.
      • Gelinas I.
      Validity of the Cognitive Behavior Driver's Inventory in predicting driving outcome.
      172Mixed6281
      Motor-free Visual Perceptual TestKorner-Bitensky et al, 2000
      • Korner-Bitensky N.A.
      • Mazer B.L.
      • Sofer S.
      • et al.
      Visual testing for readiness to drive after stroke: a multicenter study.
      269CVA6164
      TMT-B (alone)Mazer et al, 1998
      • Mazer B.L.
      • Korner-Bitensky N.
      • Sofer S.
      Predicting ability to drive after stroke.
      84CVA4648
      TMT-B (as part of GRIMPS)Austroads, 2004
      Austroads
      Model Licence Re-Assessment Procedure for Older Drivers: Stage 2 Research.
      100>80y6766
      Useful Field of View (alone)Myers et al, 2000
      • Myers R.S.
      • Ball K.K.
      • Kalina T.D.
      • Roth D.L.
      • Goode K.T.
      Relation of useful field of view and other screening tests to on-road driving performance.
      43Mixed7856
      UFOV (as part of Caltest)Austroads, 2004
      Austroads
      Model Licence Re-Assessment Procedure for Older Drivers: Stage 2 Research.
      106>80y9271
      Maze TestSnellgrove, 2005
      • Snellgrove C.A.
      Cognitive screening for the safe driving competence of older people with mild cognitive impairment or early dementia.
      115MCI7882
      Multidomain testsWood et al, 2008
      • Wood J.M.
      • Anstey K.J.
      • Kerr G.K.
      • Lacherez P.F.
      • Lord S.
      A multidomain approach for predicting older driver safety under in-traffic road conditions.
      27070–88y9170
      Stroke Driver Screening AssessmentLundberg et al, 2003
      • Lundberg C.
      • Caneman G.
      • Samuelsson S.M.
      • Hakamies-Blomqvist L.
      • Almkvist O.
      The assessment of fitness to drive after a stroke: the Nordic Stroke Driver Screening Assessment.
      97CVA7067
      SMCTest
      Computerized tests of sensorimotor and cognitive function.
      Innes et al, 2008
      • Innes C.R.
      • Jones R.D.
      • Dalrymple-Alford J.C.
      • et al.
      Sensory-motor and cognitive tests predict driving ability of persons with brain disorders.
      50Mixed9789
      P-DrivePatomella and Kottorp, 2005
      • Patomella A.-H.
      • Kottorp A.
      An evaluation of driving ability in a simulator: a good predictor of driving ability after a stroke?.
      27CVA7670
      Clock Drawing TestFreund et al, 2005
      • Freund B.
      • Gravenstein S.
      • Ferris R.
      • Burke B.L.
      • Shaheen E.
      Drawing clocks and driving cars.
      119Older drivers6498
      Abbreviations: CVA, cerebrovascular accident; MCI, mild cognitive impairment; TMT-B, Trail Making Test part B.
      low asterisk Computerized tests of sensorimotor and cognitive function.
      In Australia in several jurisdictions, specialist occupational therapists trained in driving rehabilitation use the VRST-USyd as part of the clinical testing process. A large retrospective study of clients referred for a driving assessment revealed that the VRST-USyd had sound psychometric properties and was a promising screening test with a specified cutoff score.
      • Kay L.G.
      • Bundy A.
      • Clemson L.
      Predicting fitness to drive using the visual recognition slide test (USyd).
      The test yielded a sensitivity of 81% and specificity of 89%. In the light of this research, the VRST-USyd was updated and shortened and renamed DriveSafe. The retrospective study also identified that awareness of driving ability was an important factor contributing to safe driving. Because no test was available to measure driving awareness, the DriveAware (a driving awareness questionnaire) was developed and trialed. Although DriveAware had good construct validity, more items were needed to assess the least aware drivers to improve responsivity to change.
      • Kay L.G.
      • Bundy A.
      • Clemson L.
      The validity, reliability and predictive accuracy of the Driving Awareness Questionnaire.
      Subsequently, we modified DriveAware to address these concerns.
      The purpose of this study was firstly to examine the psychometric properties of DriveSafe and the modified version of DriveAware using Rasch modeling and secondly to examine the predictive validity of these tests used together when compared with the criterion standard.

      Methods

      Study Design

      A prospective design was used in which all clients referred to 2 driving rehabilitation centers in Sydney, Australia, in a 6-month period were recruited. Ethics approval was sought and granted by the ethics committees of both centers.

      Participants

      Consecutive clients who were eligible for inclusion in the study were asked to provide informed consent when the purpose of the study was explained. All participants had a driver's license, and their vision met the standards of the licensing authority; they also spoke and understood English. Of the 122 clients referred for a driving assessment, 6 were excluded; 5 of these required an interpreter, and the sixth had expressive aphasia. This exclusion group was predominantly male (83%) and was younger with an age range of 46 to 61 years (mean, 54y). Only 1 male client declined to participate in the study; he had hand injuries. Therefore, a total of 115 clients participated with ages ranging from 16 to 95 years (mean ± SD, 62.2±17.8). Ninety-one participants (79%) were male. Participants had a variety of diagnoses (table 2). Most (71%) participants were assessed at Calvary Rehabilitation and Geriatric Service; the remaining participants were assessed at Driver Rehabilitation and Fleet Safety Services at the University of Sydney.
      Table 2Participant Diagnoses (n=115)
      DiagnosisNumber%
      Neurologic conditions (including cerebrovascular accident, Parkinson's disease, multiple sclerosis)3934
      Dementia (including Alzheimer disease, vascular dementia, and mild cognitive impairment)3026
      Orthopedic or spinal injuries2724
      Miscellaneous conditions (including cancer, polio, vision deficits, and psychiatric disorders)1210
      Acquired brain injury76

      Procedures

      All participants completed a standard off- and on-road driving assessment. The 2-hour off-road assessment consisted of an interview; assessment of vision, physical, and cognitive function; and tests of knowledge of current road rules in addition to DriveSafe and DriveAware.

      Measures

      DriveSafe

      DriveSafe is a series of 13 images of the same roundabout (ie, rotary, 4-way intersection) projected on a screen to simulate the view through a windshield in which the number and position of pedestrians and vehicles vary (fig 1). Participants are asked to observe each image for 3 seconds and, when the image has been removed from the screen, to report details about the position and direction of travel of each pedestrian and vehicle in the image. The images vary in complexity, requiring participants to report from 4 to 16 elements. The participants complete 3 practice images to ensure that they understand the instructions. Performance is recorded as a score out of 140. The test takes 20 minutes to administer, and verbal responses (or hand gestures) are recorded by the clinician. The test was administered according to the standard instructions by using the standard scoring sheet.

      DriveAware

      DriveAware consists of a series of 8 questions in which the participant's response is compared with the clinician's rating. The questions are as follows: (1) Why have you been referred for a driving assessment? (2) Do you have any concerns about your driving? (3) Are you anxious about the driving assessment process? (4) How would you rate your driving performance now compared with 10 years ago? (5) Do you have difficulty planning who has right of way at intersections? (6) How often do you get surprised by vehicles or pedestrians “appearing out of nowhere”? (7) Do you have difficulty remembering things now? and (8) How do you think you performed on DriveSafe today?
      The participant's response for each question is scored on a defined scale from 1 (very aware) to 3 (very unaware). Similarly, by using information provided in the referral and the participant's performance on clinical tests, clinicians award scores by using a scale from 1 (client has poor performance or knowledge) to 3 (client has good performance or knowledge). An item-by-item discrepancy score is calculated by subtracting the clinician's score from the participant's score. Summing the item discrepancy scores yields a total discrepancy score, and the participant's awareness of driving ability is rated according to the following criteria: discrepancy score of 1 or less (intact awareness), discrepancy score of 2 to 4 (partial awareness), and discrepancy score of 5 to 10 (absent awareness).

      On-road assessment

      A 60-minute on-road driving assessment was completed by the same clinician within 1 week of the clinical assessment. The vehicle had automatic or manual transmission depending on the client's preference, power steering, and dual brakes. A professional driving instructor, in the passenger seat, gave directions and monitored safety while a registered driver-trained occupational therapist, sitting behind the instructor, recorded the participant's driving performance. The use of 2 assessors in the vehicle is standard clinical practice in Australia and elsewhere.
      • Fox G.
      • Bowden S.
      • Smith D.
      On-road assessment of driving competence after brain impairment: review of current practice and recommendations for a standardized examination.
      The on-road assessment began in quiet suburban streets to allow the participant to become familiar with the vehicle before progressing to more demanding driving environments. Each center used a standard route. The occupational therapist recorded and scored participant performance for each maneuver against the standard safety requirements of the licensing authority. Any driving instructor interventions, such as use of the dual brake or taking the steering wheel to prevent an accident, were recorded. After 20 minutes, the clinicians provided feedback to the participant about driving performance including positive aspects of driving performance and, if necessary, up to 2 changes needed to meet licensing standards. At the conclusion of the on-road assessment, the driving instructor and occupational therapist independently recorded the participant's level of awareness during the on-road assessment and the outcome of the assessment. Errors were classified into observation, vehicle positioning, speed control, vehicle control or planning, and judgment categories. Consistent with standard clinical practice in Australia, the outcome of the assessment communicated to the participant was an agreed decision between the driving instructor and occupational therapist. The outcome was categorized as being (1) pass: safe and legal driving and no further intervention required, (2) conditional pass: safe and legal driving with restrictions on license (eg, automatic vehicle only, limiting driving distance, time), (3) downgraded to a learner's license: to undertake a series of driving lessons to learn improved driving techniques and/or to use modifications; or (4) fail: failed to meet criteria for safe and legal driving and judged not to have the potential for improvement.
      The criteria for failure were errors in all categories of driving or substantial errors in 2 or 3 categories and/or driving instructor intervention required to avoid a collision. For the purposes of statistical analysis, the outcome categories were collapsed into pass (pass and conditional pass), intervention (downgrade to learner's license), or fail categories. Those categorized as pass were able to continue driving legally.
      A total of 8 occupational therapists conducted the assessments, 3 at Calvary Rehabilitation and Geriatric Service and 5 at Driver Rehabilitation and Fleet Safety Services, working consistently with 3 driving instructors, all of whom worked at both centers. All occupational therapists had completed graduate training in driving assessment and rehabilitation and were registered with the state-licensing authority to conduct driving assessments. Similarly, the driving instructors were qualified and registered and had completed additional training in rehabilitation techniques.

      Statistical Analysis

      Construct validity and internal reliability of DriveSafe and DriveAware were examined by using Rasch modeling
      • Bond T.G.
      • Fox M.C.
      Applying the Rasch Model: Mahwah.
      with the Winsteps Version 3.65.0 computer program.
      • Linacre J.M.
      A user's guide to WINSTEPS Ministep Rasch-Model computer programs.
      ,
      Winsteps, PO Box 811322, Chicago, IL 60681-1322.
      Rasch modeling constructs a linear measure from ordinal scores (by converting raw scores into scaled scores) and assesses the goodness of fit of both items and participants along a continuum. An item and participant map is generated in which the items are arranged in order of difficulty and participants are arranged in order of competence.
      The program generates 2 pairs of goodness of fit statistics, infit and outfit, expressed as MnSq and standardized fit statistics, which indicate how well the data from each item and participant conform to the assumptions of the Rasch model (ie, easy items are easy for all people and more competent people perform better on all items). For adequate fit to the model, MnSq values of 1.0±0.5 and standardized values of −2 to 2 were taken as acceptable for items and participants.
      • Linacre J.M.
      A user's guide to WINSTEPS Ministep Rasch-Model computer programs.
      Fit statistics below and above this level indicates too little or too much variation respectively. Those items and people with fit statistics over the acceptable level should be considered for removal from the test.
      • Garratt A.M.
      Rasch analysis of the Roland Disability Questionnaire.
      Ninety-five percent fit is desired and provides evidence of a single underlying construct (ie, unidimensionality).
      • Linacre J.M.
      A user's guide to WINSTEPS Ministep Rasch-Model computer programs.
      Additionally, point measure correlation coefficients should be positive and large enough to show a strong relationship between the item and the construct.
      • Streiner D.
      • Norman G.
      Health measurement scales: a practical guide to their development and use.
      A principal contrasts analysis was also conducted to examine the unidimensionality of the tests. When the empirical variance closely matches the modeled variance and when the unexplained variance from the first factor is less than 3 Eigenvalue units, then the test fits the expectations of the Rasch model, providing additional evidence that the test is unidimensional.
      • Linacre J.M.
      A user's guide to WINSTEPS Ministep Rasch-Model computer programs.
      Rasch modeling also produces reliability estimates for both items and participants. A separation statistic provides evidence of internal reliability or the ability of the test to separate groups of participants into levels of ability. To conclude that differences in the measure are caused by real differences in the extent to which participants possess the trait and not the error of measurement, the separation statistic should be 2.00 or greater.
      • Linacre J.M.
      A user's guide to WINSTEPS Ministep Rasch-Model computer programs.
      The participant reliability index (Cronbach α equivalent) and the item reliability index or replicability of placement of participants or items along the continuum should be .80 or better.
      • Linacre J.M.
      A user's guide to WINSTEPS Ministep Rasch-Model computer programs.
      To ensure that the test items function similarly for participants with different relevant characteristics (eg, sex and age), a DIF analysis was computed by Winsteps.
      • Badia X.
      • Prieto L.
      • Linacre J.M.
      Differential item and test functioning.
      Items that were functioning significantly differently (P≤.05) for participants with specific characteristics were identified and could be considered for removal from the test. When more than 1 rater is scoring the test, it is possible to use DIF analysis of raters to determine if there are any unexpectedly large differences in the way in which raters score each item on the test. This analysis provided detailed evidence for interrater reliability.
      Finally, the predictive validity of DriveSafe and DriveAware was examined. Data from participants were allocated randomly to 2 groups. Applying both tests serially, an optimal lower and upper cutoff score was determined for 1 group by using descriptive statistics, and the accuracy of these scores was then tested in the validation group to reduce the risk of capitalizing on chance. A 4-way matrix (table 3) displays these statistical concepts. Sensitivity, calculated as a/(a+c), refers to the test's ability to identify a problem when a problem truly exists (true positives) or, in this case, a failed test result and unsafe driving. Specificity, calculated as d/(b+d), refers to the test's ability to obtain a negative result when there is no problem (true negatives) or, in this case, a pass on the test and safe driving performance. The positive predictive value is the proportion of people that is identified by the test as likely to be unsafe who are actually unsafe on-road, calculated as a/(a+b). The negative predictive value is the proportion of those identified by the test as safe who are actually safe on-road, calculated as d/(c+d). Data from a perfect screening test would yield descriptive statistics of 100%. With the lower cutoff score, we were most interested in identifying those who were unsafe (ie, positive predictive value) and minimizing the proportion of drivers falsely categorized as unsafe (ie, specificity). But with the upper cutoff score, we were most interested in identifying those who were safe (ie, negative predictive value) and minimizing the proportion of drivers falsely categorized as safe (ie, sensitivity). A CI of 95% for sensitivity and specificity was calculated by using the Agresti-Coull method for 2 samples.
      • Brown L.D.
      • Cai T.T.
      • DasGupta A.
      Interval estimation for a binomial proportion.
      Table 3Matrix for Determining Sensitivity, Specificity, and Positive and Negative Predictive Values
      Actual Driving Performance
      Screening Test ResultsUnsafeSafe
      Unsafe (fail test)a (true-positive)b (false-positive)
      Safe (pass test)c (false-negative)d (true-negative)

      Results

      DriveSafe

      Each of the 13 DriveSafe images was scored as 2 test items; the first item represented the objects to be recalled in each image (ie, pedestrians and/or vehicles), and the second item included the details of each object to be recalled (eg, location on the road and direction of travel). Preliminary analysis revealed that images 11 and 13 had fit statistics beyond the acceptable range (ie, outfit MnSq of 2.05 and 2.33, respectively). Further analysis revealed that there was a large discrepancy in the scores for these 2 images for a small number of people. These items also had low point measure correlation coefficients (ie, .15 and .13, respectively) indicating that the items were not closely related to the construct. Therefore, these 2 test items were excluded. After these deletions, the final test included 11 images generating 22 items.
      All of the remaining items had infit and outfit statistics within the acceptable range. The map of items and drivers (fig 2) showed the spread of driver ability from the more capable to less capable in SD units. Except for the most competent drivers, the range of item difficulty was comparable to the range of drivers' ability. Those drivers who were least competent were assessed well, and this is the group of greatest concern. Point measure correlation coefficients were all positive and ranged from .30 to .89 (mean, .61), supporting the validity of the measure. Data from 97% of the participants fit the model, providing evidence for the unidimensionality of the measure.
      Figure thumbnail gr2
      Fig 2Map of drivers and items for DriveSafe. Abbreviations: M, mean; S, 1 SD; T, 2 SDs; X, 1 person.
      The principle contrast analysis revealed that the modeled variance was high (79%) and closely matched the empirical variance (79.5%). Furthermore, the unexplained variance by the first contrast was only 2.2 Eigenvalue units. These statistics provide further evidence that DriveSafe reflects a unidimensional construct.
      The test separated the participants into 4 groups (a model separation of 4.6) with a participant reliability index (Cronbach α equivalent) of .95. The item reliability index was .97. These results provide evidence for DriveSafe's internal reliability.
      The DIF analysis revealed that there were no significant differences (t<1.96, P>.05) in the performance on DriveSafe associated with sex. For the majority of items, there were no significant differences in performance on the test based on age. When the DIF analysis was applied to raters, there was no significant difference (t<1.96, P>.05) in how all raters scored all test items. Thus, the DIF analysis provided strong evidence for the interrater reliability of DriveSafe on an item-by-item level.

      DriveAware

      Preliminary analysis with Winsteps revealed that item 4, comparing current driving performance with performance 10 years ago, failed to fit the Rasch model, and, on reflection, we decided it should be excluded because it was too difficult for people to make this comparison. The discrepancy score for each item was rated on a 5-point ordinal scale (ie, a discrepancy of –2 was scored as a 0; a discrepancy of 2 was scored as a 4). In this sample, the total scores on DriveAware ranged from 11 to 21. Item 3 (awareness of implications of a driving assessment) and item 7 (awareness of cognitive deficits) showed category structure problems and were thus rescored by collapsing the scoring to a 4-category scale (0, 1, 2, 3). When these adjustments were made, both the infit and outfit statistics for most items were within the acceptable range. However, fit statistics for item 6, awareness of hazards, were just beyond the acceptable range. The point measure correlation coefficients were all positive and ranged from .57 to .82 (mean, .71), supporting the validity of the test. Ninety-three percent of participants fitted the assumptions of the Rasch model. Furthermore, the principal contrasts analysis yielded a modeled variance (71.8%) that closely matched the empirical variance (71.7%), and the unexplained variance by the first contrast was 1.8 Eigenvalue units, providing evidence that DriveAware measures a unidimensional construct.
      The map (fig 3) shows the spread of drivers' awareness (from intact to absent) with test item difficulty. The map reveals gaps in the test in which there are no items to measure drivers' awareness. This distribution of items suggests that DriveAware is not a stand-alone test but should be used in conjunction with another test. Nonetheless, the hierarchy of items generated by Rasch analysis was conceptually logical, reflecting a progression of awareness. Thus, being aware of deficits in performance on DriveSafe was easier than being aware of deficits in complex driving situations, and it was hardest for participants to show awareness of deficits in general driving performance that should cause concern.
      Figure thumbnail gr3
      Fig 3Map of drivers and items for DriveAware. Abbreviations: M, mean; S, 1 SD; T, 2 SDs; #, 2 people; *, 1 person.
      The test separated drivers into 2 groups (absent and intact, a model separation of 2.05), suggesting that partial awareness is not a useful category. The participant reliability index (Cronbach α equivalent) was .81, indicating a moderately high replicability of the ordering of participants on the hierarchy.
      • Bond T.G.
      • Fox M.C.
      Applying the Rasch Model: Mahwah.
      The item reliability index was very high (.99). Together, these results provide strong evidence for the internal reliability of DriveAware.
      The DIF analysis revealed no significant differences (t<1.96, P>.05) in test performance based on sex. There was a significant difference (t<1.96, P≤.05) on 1 test item, awareness of hazards in the driving environment, between novice (aged 16–25y) and older drivers (aged ≥66y). When DIF analysis was applied to raters, there were significant differences (P≤.05) in how 4 raters scored 1 item and how 2 of these raters scored a further 2 items. These differences do not necessarily reflect differences between raters, although that is a possibility; it may also reflect real differences among participants. Nonetheless, there was substantial evidence for the interrater reliability at an item-by-item level for most raters.

      Predictive Validity of DriveSafe and DriveAware

      A total of 45 participants (39%) passed the driving assessment, whereas 36 participants (31%) failed. Thirty-four participants required driving lessons to learn to use modifications or to improve driving performance; 29 passed a reassessment.
      To calculate predictive validity, DriveSafe and DriveAware were used together in a process of trichotomization to identify participants who were predicted to fail, those who were predicted to pass, and those who needed to complete an on-road assessment. Those participants who were learner drivers or who primarily had physical deficits (n=19) were excluded from this calculation because they needed to complete an on-road assessment to determine appropriate modifications. To include them would have overestimated the accuracy of the prediction.
      The optimal lower cutoff score was 76 or less (out of 128) on DriveSafe and higher than 17 on DriveAware, and the optimal upper cutoff score was more than 95 (out of 128) on DriveSafe and less than 15 on DriveAware. The numbers of participants correctly and incorrectly classified with these cutoff scores are reported in Table 4, Table 5. The sensitivity, specificity, and positive and negative predictive values for the optimal cutoff for the test (group 1) and validation groups (group 2) are presented in table 6.
      Table 4Classification of Participants Using Optimal Lower Cutoff Score for Combined Tests
      DriveSafe and DriveAware ResultsActual Driving Performance
      Group 1 (n=52)Group 2 (n=44)
      Unsafe (Fail)Safe (Pass)Unsafe (Fail)Safe (Pass)
      Unsafe (fail test)81131
      Safe (pass test)1132426
      Table 5Classification of Participants by Using Optimal Upper Cutoff Score for Combined Tests
      DriveSafe and DriveAware ResultsActual Driving Performance
      Group 1 (n=52)Group 2 (n=44)
      Unsafe (Fail)Safe (Pass)Unsafe (Fail)Safe (Pass)
      Unsafe (fail test)27112112
      Safe (pass test)212110
      Table 6Descriptive Statistics for Optimal Lower- and Upper-Cutoff Scores of Combined Tests
      Lower CutoffUpper Cutoff
      Group 1 (n=52)Group 2 (n=44)Group 1 (n=52)Group 2 (n=44)
      Sensitivity42 (23–64)76 (52–91)93 (77–99)95 (76–100)
      Specificity97 (83–100)96 (80–100)52 (32–71)45 (27–65)
      Positive predictive value89937164
      Negative predictive value74878691
      NOTE. Values are in percentages or percentages (95% CI).
      For the lower cutoff, the estimated difference in specificity for the test and validation sample was 1% (95% CI, –11% to 13%). Of less importance, the estimated difference in sensitivity for the 2 groups was –34% (95% CI, –60% to –2%). For the upper cutoff, the estimated difference in sensitivity for the test and validation samples was –2% (95% CI, –17% to 14%). The estimated difference in specificity for the 2 groups was –7% (95% CI, –22% to 34%). Thus, there is no evidence to suggest there is any difference in the test and validation samples although the sample size is small.
      Furthermore, with this lower cutoff, the positive predictive values of 89% and 93% indicated that drivers predicted to be unsafe had a high probability of being unsafe. The negative predictive values generated by the upper cutoff (86% and 91%, respectively) indicated that drivers predicted to be safe had a moderately high probability of being safe. However, it is important to note that these values are dependent on the prevalence of unsafe or safe driving in the sample. If the sample included higher numbers of unsafe or safe drivers, then the positive and negative predictive values might change. Using the upper and lower cutoff scores, 48 drivers with cognitive impairments were identified as requiring further testing.
      The process of trichotomization is represented in a flow chart in figure 4. Thus, drivers with cognitive impairment who scored 76 or less on DriveSafe and higher than 17 on DriveAware were categorized as unsafe and should cease driving, those who scored higher than 95 on DriveSafe and less than 15 on DriveAware were categorized as safe and could continue to drive, and those whose test results fell between these cutoff scores need to proceed to an on-road assessment to clarify their driving safety.
      Figure thumbnail gr4
      Fig 4Flowchart for the trichotomization process.

      Discussion

      This research has important implications for clinical practice. Although driving rehabilitation specialists agree that 1 purpose of the off-road assessment is to identify drivers who would potentially benefit from an on-road assessment, almost all drivers proceed to an on-road assessment, primarily because off-road tests cannot predict accurately those who do not need further assessment.
      • Unsworth C.A.
      • Lovell R.K.
      • Terrington N.S.
      • et al.
      Review of tests contributing to the occupational therapy off-road driver assessment.
      The promising results of this study suggest that it is possible to predict with substantial accuracy those with cognitive impairment who do not need an on-road assessment. In so doing, scarce clinical resources would be optimized for those who would benefit most. Additionally, drivers' resources could be used more appropriately to explore alternative transport options or other pertinent issues.
      Before we could test the sensitivity and specificity of the cutoff scores for the 2 tests, when used together to predict on-road performance, we needed to examine the psychometric properties of DriveSafe and DriveAware separately after they were modified.
      • Kay L.G.
      • Bundy A.
      • Clemson L.
      Predicting fitness to drive using the visual recognition slide test (USyd).
      • Kay L.G.
      • Bundy A.
      • Clemson L.
      The validity, reliability and predictive accuracy of the Driving Awareness Questionnaire.
      The findings yielded strong evidence for the construct validity and internal reliability of DriveSafe that measures a single construct, namely awareness of the driving environment. Similarly, the results showed strong evidence for the construct validity and internal reliability of DriveAware. It measured the theoretic construct, awareness of driving ability, and separated drivers into 2 categories: those with and without awareness. Although awareness is theoretically a continuous construct,
      • Pachana N.A.
      • Petriwskyj A.M.
      Assessment of insight and self-awareness in older drivers.
      Rasch analysis of DriveAware suggested that in the clinical setting DriveAware can measure reliably only the presence or absence of the construct.
      These findings are consistent with the theoretic driving literature. Driving is conceptualized as a hierarchical task influenced by adequate knowledge and skills to interpret the driving environment.
      • Hoeschen A.
      • Bekiaris E.
      Deliverable No 2.1: inventory of driver training needs and major gaps in the relevant training procedures (Trainer GRD1-1999-10024).
      • Pirenne D.
      • Arno P.
      • Baten G.
      • et al.
      Deliverable No 5.1: Trainer assessment criteria and methodology (Trainer GRD1-1999-10024).
      Our results indicate that DriveSafe provides information about drivers' global awareness of the driving environment. Most other tests used in driving research assess component visual processing (eg, Motor-free Visual Perception Test
      • Calarusso R.P.
      • Hammill D.D.
      Motor-Free Visual Perception Test.
      ), cognition (eg, Mini-Mental Status Examination
      • Folstein M.F.
      • Folstein S.E.
      • McHugh P.R.
      “Mini-mental state.” A practical method for grading the cognitive state of patients for the clinician.
      ), executive function (eg, Clock Drawing Test
      • Freund B.
      • Gravenstein S.
      • Ferris R.
      • Burke B.L.
      • Shaheen E.
      Drawing clocks and driving cars.
      and Trail Making Tests
      • Reitan R.M.
      Trail Making Test: manual for administration, scoring and interpretation.
      ), or some combination of these (eg, Useful Field of View
      • Myers R.S.
      • Ball K.K.
      • Kalina T.D.
      • Roth D.L.
      • Goode K.T.
      Relation of useful field of view and other screening tests to on-road driving performance.
      ). DriveSafe is unique because it does not deconstruct driving into component skills. DriveAware enables clinicians to determine whether drivers are aware of their driving ability without observing their driving performance. Thus, using both DriveSafe and DriveAware together enables the prediction of driving performance by measuring drivers' awareness of both the driving environment and their own skills.
      Once evidence for the valid and reliable measurement was established, we examined the predictive validity of DriveSafe and DriveAware together. This was measured by the ability of the tests to trichotomize drivers with cognitive impairments referred for a driving assessment compared with a standard on-road assessment (ie, fail, pass, require further testing). By using these optimal lower cutoff scores of 76 or less on DriveSafe and more than 17 on DriveAware and upper cutoff scores of more than 95 on DriveSafe and less than 15 on DriveAware, drivers were categorized with accuracy unrivaled by any other published assessments.
      When using the 2 tests together, 50% of drivers were predicted to be either safe or unsafe without an on-road assessment, and 90% were correctly classified. Examination of the errors of classification revealed that the 2 drivers who were predicted to be unsafe but who passed the on-road test had a diagnosis of dementia. Drivers with dementia require assessment at least every 6 months because of the progression of the disease.
      • Dubinsky R.M.
      • Stein A.C.
      • Lyons K.
      Practice parameter: risk of driving and Alzheimer's disease (an evidence-based review): report of the quality standards subcommittee of the American Academy of Neurology.
      • Hecker J.
      • Snellgrove C.A.
      Australian Society for Geriatric Medicine Position Statement No. 11: Driving and dementia.
      • Molnar F.J.
      • Patel A.
      • Marshall S.C.
      • Man-Son-Hing M.
      • Wilson K.G.
      Systematic review of the optimal frequency of follow-up in persons with mild dementia who continue to drive.
      Consequently, it is likely that our testing anticipated the revoking of these 2 drivers' licenses by 6 months. The other 3 errors of classification were drivers predicted to be safe. All 3 required driving lessons to improve driving behavior and went on to be safe. Determining the level of error that is acceptable when using screening tests is a cost-benefit judgment based on the impact of incorrect classification.
      • Portney L.G.
      • Watkins M.P.
      Foundations of clinical research: applications to practice.
      For the driver, any error in classification is unsatisfactory, but researchers attempt to determine the risk threshold that is acceptable to society.
      • Molnar F.J.
      • Patel A.
      • Marshall S.C.
      • Man-Son-Hing M.
      • Wilson K.G.
      Clinical utility of office-based cognitive predictors of fitness to drive in persons with dementia: a systematic review.
      Nonetheless, there is a substantial cost benefit in conducting on-road assessments for only 50% of drivers with cognitive impairment referred to the driving rehabilitation services; these figures are supported by DriveSafe and DriveAware.
      DriveSafe and DriveAware also have the advantage of being generic tests that are applicable to a wide range of diagnostic groups. When most driving rehabilitation centers assess drivers with diverse diagnoses, having 2 tests that can be used with all clients streamlines clinical procedures.
      • Wood J.M.
      • Anstey K.J.
      • Kerr G.K.
      • Lacherez P.F.
      • Lord S.
      A multidomain approach for predicting older driver safety under in-traffic road conditions.

      Study Limitations

      This research had several limitations. The sample size was small, although it was larger than many samples in similar literature. The sample also represented a self-selected group who had been referred for and underwent a driving assessment. Although it is possible that some drivers who thought they would fail the assessment would not have proceeded with it, the substantial number of drivers who did fail suggests that this was not the case. The on-road assessors were not masked to the off-road assessment results, but they were unaware of the cutoff scores used for categorizing drivers. The promising results need to be verified in a larger study before being implemented in clinical settings. Furthermore, the tests themselves have inherent limitations being reliant on verbal responses and requiring a trained administrator. The future development of the tests in a computerized format may address these issues without compromising the psychometric properties, but this requires further research.

      Conclusions

      This study is potentially important for both clinicians and the community. As the population ages and the proportion of older people who continue to drive increases, there will be a growing need to identify drivers who are at risk. The demand for driving assessment and rehabilitation services will increase. Therefore, the possibility of accurately identifying drivers who are safe or unsafe or who need further testing is an encouraging development. By assessing global awareness of the driving environment together with awareness of driving ability, DriveSafe and DriveAware break new ground. Further research is necessary to verify the promising results of this study.
      Supplier
      aWinsteps, PO Box 811322, Chicago, IL 60681-1322.

      Acknowledgments

      We acknowledge the assistance of clinicians and clients at Driver Rehabilitation and Fleet Safety Services and Calvary Rehabilitation and Geriatric Services in conducting this research.

      References

        • Mazer B.
        • Gelinas I.
        • Benoit D.
        Evaluating and retraining driving performance in clients with disabilities.
        Crit Rev Phys Rehabil Med. 2004; 16: 291-326
        • Alvarez F.J.
        • Fierro I.
        Older drivers, medical condition, medical impairment and crash risk.
        Accid Anal Prev. 2008; 40: 55-60
        • Ponsford A.S.
        • Viitanen M.
        • Lundberg C.
        • Johansson K.
        Assessment of driving after stroke—a pluridisciplinary task.
        Accid Anal Prev. 2008; 40: 452-460
        • Akinwuntan A.E.
        • Feys H.
        • De Weerdt W.
        • Baten G.
        • Arno P.
        • Kiekens C.
        Prediction of driving after stroke: a prospective study.
        Neurorehabil Neural Repair. 2006; 20: 417-423
        • Freund B.
        • Gravenstein S.
        • Ferris R.
        • Burke B.L.
        • Shaheen E.
        Drawing clocks and driving cars.
        J Gen Intern Med. 2005; 20: 240-244
        • Owsley C.
        Clinical and research issues on older drivers: future directions.
        Alzheimer Dis Assoc Disord. 1997; 11: 3-7
        • Molnar F.J.
        • Patel A.
        • Marshall S.C.
        • Man-Son-Hing M.
        • Wilson K.G.
        Clinical utility of office-based cognitive predictors of fitness to drive in persons with dementia: a systematic review.
        J Am Geriatr Soc. 2006; 54: 1809-1824
        • Innes C.R.
        • Jones R.D.
        • Dalrymple-Alford J.C.
        • et al.
        Sensory-motor and cognitive tests predict driving ability of persons with brain disorders.
        J Neurol Sci. 2007; 260: 188-198
        • Portney L.G.
        • Watkins M.P.
        Foundations of clinical research: applications to practice.
        2nd ed. Prentice Hall Health, Upper Saddle River2000
        • Myers R.S.
        • Ball K.K.
        • Kalina T.D.
        • Roth D.L.
        • Goode K.T.
        Relation of useful field of view and other screening tests to on-road driving performance.
        Percept Mot Skills. 2000; 91: 279-290
        • Bouillon L.
        • Mazer B.
        • Gelinas I.
        Validity of the Cognitive Behavior Driver's Inventory in predicting driving outcome.
        Am J Occup Ther. 2006; 60: 420-427
        • Snellgrove C.A.
        Cognitive screening for the safe driving competence of older people with mild cognitive impairment or early dementia.
        Australian Transport Safety Bureau, Canberra2005
        • Reitan R.M.
        Trail Making Test: manual for administration, scoring and interpretation.
        Indiana University Medical Center, Indianapolis1958
        • Mazer B.L.
        • Korner-Bitensky N.
        • Sofer S.
        Predicting ability to drive after stroke.
        Arch Phys Med Rehabil. 1998; 79: 743-750
        • Austroads
        Model Licence Re-Assessment Procedure for Older Drivers: Stage 2 Research.
        (Sydney, Report No.: AP-R259/04)2004
        • Calarusso R.P.
        • Hammill D.D.
        Motor-Free Visual Perception Test.
        Western Psychological Services, Los Angeles1972
        • Korner-Bitensky N.A.
        • Mazer B.L.
        • Sofer S.
        • et al.
        Visual testing for readiness to drive after stroke: a multicenter study.
        Am J Phys Med Rehabil. 2000; 79: 253-259
        • Lundberg C.
        • Caneman G.
        • Samuelsson S.M.
        • Hakamies-Blomqvist L.
        • Almkvist O.
        The assessment of fitness to drive after a stroke: the Nordic Stroke Driver Screening Assessment.
        Scand J Psychol. 2003; 44: 23-30
        • Wood J.M.
        • Anstey K.J.
        • Kerr G.K.
        • Lacherez P.F.
        • Lord S.
        A multidomain approach for predicting older driver safety under in-traffic road conditions.
        J Am Geriatr Soc. 2008; 56: 986-993
        • Patomella A.-H.
        • Kottorp A.
        An evaluation of driving ability in a simulator: a good predictor of driving ability after a stroke?.
        Third International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design. Maine, Portland2005 (p 104-10)
        • Kay L.G.
        • Bundy A.
        • Clemson L.
        Predicting fitness to drive using the visual recognition slide test (USyd).
        Am J Occup Ther. 2008; 62: 187-197
        • Kay L.G.
        • Bundy A.
        • Clemson L.
        The validity, reliability and predictive accuracy of the Driving Awareness Questionnaire.
        Disabil Rehabil. 2009; 11: 1-9
        • Fox G.
        • Bowden S.
        • Smith D.
        On-road assessment of driving competence after brain impairment: review of current practice and recommendations for a standardized examination.
        Arch Phys Med Rehabil. 1998; 79: 1288-1296
        • Bond T.G.
        • Fox M.C.
        Applying the Rasch Model: Mahwah.
        L. Erlbaum, NJ2001
        • Linacre J.M.
        A user's guide to WINSTEPS Ministep Rasch-Model computer programs.
        (2008.Accessed May 7, 2008)
        • Garratt A.M.
        Rasch analysis of the Roland Disability Questionnaire.
        Spine. 2003; 28: 79-84
        • Streiner D.
        • Norman G.
        Health measurement scales: a practical guide to their development and use.
        Oxford Univ, Oxford1995
        • Badia X.
        • Prieto L.
        • Linacre J.M.
        Differential item and test functioning.
        Rasch Measurement Transactions. 2002 (p. 889) (Accessed September 26, 2006)
        • Brown L.D.
        • Cai T.T.
        • DasGupta A.
        Interval estimation for a binomial proportion.
        Stat Sci. 2001; 16: 101-133
        • Unsworth C.A.
        • Lovell R.K.
        • Terrington N.S.
        • et al.
        Review of tests contributing to the occupational therapy off-road driver assessment.
        Aust Occup Ther J. 2005; 52: 57-74
        • Pachana N.A.
        • Petriwskyj A.M.
        Assessment of insight and self-awareness in older drivers.
        Clin Gerontol. 2006; 30: 23-38
        • Hoeschen A.
        • Bekiaris E.
        Deliverable No 2.1: inventory of driver training needs and major gaps in the relevant training procedures (Trainer GRD1-1999-10024).
        European Commission, Competitive and Sustainable Growth Programme, Directorate General for Energy and Transport, Brussels, Belgium2001
        • Pirenne D.
        • Arno P.
        • Baten G.
        • et al.
        Deliverable No 5.1: Trainer assessment criteria and methodology (Trainer GRD1-1999-10024).
        European Commission, Competitive and Sustainable Growth Programme, Directorate General for Energy and Transport, Brussels, Belgium2002
        • Folstein M.F.
        • Folstein S.E.
        • McHugh P.R.
        “Mini-mental state.” A practical method for grading the cognitive state of patients for the clinician.
        J Psychiatr Res. 1975; 12: 189-198
        • Dubinsky R.M.
        • Stein A.C.
        • Lyons K.
        Practice parameter: risk of driving and Alzheimer's disease (an evidence-based review): report of the quality standards subcommittee of the American Academy of Neurology.
        Neurology. 2000; 54: 2205-2211
        • Hecker J.
        • Snellgrove C.A.
        Australian Society for Geriatric Medicine Position Statement No. 11: Driving and dementia.
        Australas J Aging. 2003; 22: 46-50
        • Molnar F.J.
        • Patel A.
        • Marshall S.C.
        • Man-Son-Hing M.
        • Wilson K.G.
        Systematic review of the optimal frequency of follow-up in persons with mild dementia who continue to drive.
        Alzheimer Dis Assoc Disord. 2006; 20: 295-297