If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Department of Rehabilitation Sciences, University of British Columbia, Vancouver, CanadaInternational Collaboration on Repair Discoveries, Vancouver, Canada
Lui J, MacGillivray MK, Sawatzky BJ. Test-retest reliability and minimal detectable change of the SmartWheel clinical protocol.
Objectives
(1) To quantify the intra- and intersession reliability and minimal detectable change (MDC) of the SmartWheel clinical protocol (SCP) and (2) to compare the reliability of the SCP between experienced and naïve wheelchair users.
Design
Test-retest study.
Setting
Biomechanics laboratory.
Participants
Manual wheelchair users (WCUs) (n=10) with 1 to 32 years of wheeling experience and able-bodied (AB) naïve users (n=15).
Interventions
Not applicable.
Main Outcome Measures
Wheelchair propulsion parameters including average peak force, push frequency, push length, and velocity were measured according to the SCP but with 5 trials per session, for 2 sessions. WCUs and AB users were analyzed separately. Intraclass correlation coefficients ([ICC]2,1 and ICC2,5) were calculated to assess intrasession reliability. ICC2,1 and MDC (with 95% confidence) were calculated for each SCP parameter using a single trial from each session and with the mean of 5 repeated measures to evaluate intersession reliability.
Results
Intra- and intersession reliability for WCU parameters ranged from high to very high correlation (ICC range, .70–.99). For AB parameters, intrasession ICC2,1 ranged from moderate to very high (ICC range, .50–.92), while intersession ICC2,1 indicated low to very high correlation (ICC range, .25–.90). Estimates of standard error of measurement and MDC were provided for each parameter. For both WCUs and AB users, using the means from 5 trials increased intra- and intersession ICC and decreased MDC values. All MDC values were lower for WCUs compared with AB users.
Conclusions
The SCP is a reliable method for assessing propulsion parameters in WCUs, even if just 1 trial is taken per session. AB users showed lower intra- and intersession reliability compared with WCUs. Therefore, for AB users or individuals with minimal wheeling experience, averaging multiple trials is recommended for the SCP.
Wheelchair configurations and interventions designed to modify propulsion strategies can effectively reduce the forces needed to propel a wheelchair and therefore reduce the load on the shoulders and wrists.
To better understand manual wheelchair propulsion, the SmartWheela was developed to allow clinicians and researchers to objectively assess the forces and moments applied to the pushrim.
The SmartWheel is a specialized wheelchair wheel that measures propulsion parameters, such as the three-dimensional forces applied to the pushrim, push length, push frequency, and velocity during wheeling.
The SmartWheel is used in conjunction with the SmartWheel clinical software to assist clinicians in assessing these propulsion parameters. The SmartWheel can provide clinicians with objective data needed to prescribe the most appropriate wheelchair configurations or to implement and track the effectiveness of their interventions.
The SmartWheel users' group, an international network of researchers and clinicians, developed a SmartWheel clinical protocol (SCP), which was designed to accommodate a clinic's availability of space and time.
Preliminary outcomes of the SmartWheel User's Group database: a proposed framework for clinicians to objectively evaluate manual wheelchair propulsion.
In the SCP, patients are instructed to wheel at a self-selected speed for a maximum of 10 seconds (or 10m) on 3 different surfaces (tile, 5% grade ramp, and carpet) while the SmartWheel software generates data on 4 key measurements: velocity, average peak force, push frequency, and push length.
Preliminary outcomes of the SmartWheel User's Group database: a proposed framework for clinicians to objectively evaluate manual wheelchair propulsion.
This provides 12 SCP parameters (3 surfaces × 4 measurements), which are considered the most clinically relevant and are used to help guide wheelchair prescription and/or training.
Preliminary outcomes of the SmartWheel User's Group database: a proposed framework for clinicians to objectively evaluate manual wheelchair propulsion.
Accordingly, velocity, average peak force, push frequency, and push length are important measures that clinicians can modify in order to improve the performance or efficiency of manual wheelchair propulsion.
Consortium for Spinal Cord Medicine Preservation of upper limb function following spinal cord injury: a clinical practice guideline for health-care professionals.
Paralyzed Veterans of America,
Washington (DC)2005
Therefore, it is important that these measurements are reliably determined by the SCP. It is also important to estimate the minimal detectable change (MDC) of the protocol. This would allow clinicians to distinguish true variability from measurement error and to estimate the minimal difference that can be detected confidently between sessions.
This could mean that the reliability of the SCP may be dependent on general wheelchair experience and therefore should be evaluated accordingly.
To date, no studies have reported the test-retest reliability or the MDC of the SCP. Therefore, the aims of the study were to: (1) examine the intra- and intersession reliability of the SCP; (2) quantify the MDC for each of the SCP parameters; and (3) compare the intra- and intersession reliability between experienced and naïve WCUs.
for calculating sample size for reliability studies, it was determined that a minimum sample size of 9 was needed.
Ten spinal cord injured manual WCUs with at least 1 year of wheeling experience were recruited through advertisements posted at an outpatient rehabilitation clinic. Only WCUs whose wheelchairs were equipped with 61.0 or 63.5cm (24 or 25in). wheels, quick release axles, and pneumatic high-pressure tires were included in the study. Fifteen able-bodied (AB) individuals were also included and analyzed separately from the WCUs. The AB individuals were comprised of a convenience sample to represent individuals who are naïve to manual wheeling. Therefore, they are to some extent comparable with newly injured individuals with intact upper body function. Inclusion criteria for all participants included being between 18 to 65 years of age, and being able to use a manual wheelchair without pain. Individuals who had muscular or cardiorespiratory illnesses were excluded from this study. The study received approval from the university's clinical research ethics board. All participants provided written informed consent.
Study Design
The study used a test-retest design. All participants were evaluated with the same protocol twice, with approximately 7 days between testing sessions.
Procedure
WCUs used their own wheelchairs for the study, and AB individuals were provided with appropriately sized (either 40.6 or 43.2cm [16 or 17in]) Elevationb wheelchairs that allowed for seat height and dump adjustments via a built in hydraulic spring system. Individual wheelchair configurations were kept consistent between sessions. The SmartWheel was affixed to the wheelchair on the side of the participant's dominant arm while the wheelchair's standard wheel remained on the nondominant side. Pneumatic high-pressure tires were used on all wheels. All tires were inflated to manufacturer's specifications.
Once the SmartWheel was attached, all participants were given time to familiarize themselves with their wheelchairs and the different test surfaces prior to testing. The testing orders of the 3 surfaces were randomized. Adequate rest was provided at any time (at participant's request) during testing in order to minimize fatigue.
Protocol
Following the SCP, participants were instructed to wheel across 3 different surfaces: tile, ramp, and carpet. Data were collected in accordance with the SCP, but with additional trials for each surface. The current SCP does not specify the number of trials that should be collected. However, a total of 5 trials were collected for each surface in order to (1) evaluate the intrasession reliability and (2) evaluate the effect of averaging multiple trials on the intersession reliability.
As defined by the SCP for the tile surface, wheeling was done on smooth, level tile. Participants accelerated from a stationary position up to a comfortable, self-selected speed and wheeled continuously for 10 seconds or 10 meters, whichever occurred first. Collection started as soon as the SmartWheel pushrim was contacted and participants were told to keep wheeling at a normal speed until instructed to stop. The SCP was administered for 5 consecutive trials. The ramp and carpet protocols followed the same procedure as the tile protocol, but with a 5% grade ramp and a low-pile carpet, respectively.
Measures
Four main measurements were computed with the SmartWheel clinical software for each of the 3 surfaces: average peak force (N/kg), average push frequency (Hz), average push length (°), and average velocity (m/s). These are the 12 SCP parameters currently used by clinicians to assess their patients. Calculations for each parameter were based on the data collected from all pushes except for the first 3. This was done to capture the participants' propulsion when they are assumed to have completed the bulk of their acceleration.
Preliminary outcomes of the SmartWheel User's Group database: a proposed framework for clinicians to objectively evaluate manual wheelchair propulsion.
For each parameter, the intrasession reliability was calculated using intraclass correlation coefficients (ICCs): ICC2,1 (single measures) and ICC2,5 (average measures of 5 trials).
Both were based on a 2-way (random effects) repeated-measures analysis of variance model with absolute agreement. For each session, intrasession paired t tests were calculated between the first and fifth trials for each parameter in order to detect potential learning effects. The alpha level of .05 was adjusted using a Bonferroni correction for 12 comparisons.
The intersession reliability was calculated using ICC2,1 with the first trial of each session for each parameter. In addition, to ensure that stable measures were obtained for each wheeling parameter, the mean of 5 trials was also calculated for each parameter of each session. ICC2,1 for intersession reliability was then calculated using the mean parameter values of each session. Although there are many ways to interpret ICC values, we chose Munro's classification of reliability coefficients: .26 to .49 reflects low correlation; .50 to .69 reflects moderate correlation; .70 to .89 reflects high correlation; and .90 to 1.00 indicates very high correlation.
Standard error of measurement was calculated to assess the measurement error across sessions because of variability. Standard error of measurement is an index of absolute reliability; therefore, it estimates the precision of individual scores.
The standard error of measurement was estimated using the mean squared error (MSE) from a 2-way analysis of variance, where standard error of measurement = √MSE. This formulation has the advantage of being independent of the ICC values, and therefore is unaffected by the range of measurement values.
To estimate the minimal amount of change needed to exceed the measurement error, the MDC for each SCP parameter was estimated based on a 95% confidence interval, where MDC = standard error of measurement × √2 × 1.96.
Bland-Altman plots were constructed for each parameter by plotting the mean difference between session measurements (using the mean values of 5 trials) against the mean of the 2 test sessions for each participant.
Ninety-five percent confidence intervals (limits of agreement) were calculated to determine whether there were any systematic biases between sessions. All analyses were done using SPSSc and Microsoft Excel.d
Results
WCU Group
A total of 7 men and 3 women WCUs were recruited with a mean age of 36.2±12.3 years. Eight of the participants were persons with paraplegia and 2 were persons with tetraplegia. Wheeling experience ranged from 1 to 32 years, with a mean of 16.5 years.
Table 1 shows the means (across all trials) and SDs of all parameters for each session. ICC values for intra- and intersession reliability are shown in table 2. For WCUs, intrasession correlation was high for 7 of 12 SCP parameters and very high for the other 5 parameters. As expected, using ICC2,5 produced higher ratings; all parameters indicated very high correlation. Paired t tests showed no significant differences between the first and fifth trial for all parameters.
For intersession reliability, based on the first trials of each session, most parameters had ICC2,1 values that reflected high correlation (see table 2). Seven of 12 SCP parameters had high correlations and 5 had very high correlations (frequency, push length, and velocity on ramp, as well as force and velocity on carpet). ICC2,1 based on the mean of 5 trials resulted in very high correlations for 10 of 12 SCP parameters, while 2 parameters had high correlations (tile force and frequency). Standard error of measurement and MDC values at the 95% confidence level are provided (see table 2). Standard error of measurement and MDC values based on the mean of 5 repeated measures were lower for all parameters compared with using only the first trials. Bland-Altman plots revealed no systematic biases between sessions, for all parameters (fig 1).
Fig 1Sample Bland-Altman plots for WCUs: average peak force, push length, push frequency, and velocity. Mean differences (using the mean of 5 trials) between sessions are plotted against the mean of the 2 test sessions for each participant.
Ten men and 5 women were recruited, with a mean age of 30.3±9.4 years. All 15 AB individuals had no prior wheeling experience. Table 1 shows the test and retest means and SD for all parameters.
Using ICC2,1, intrasession correlation was moderate for 4 of 12 SCP parameters; 6 of 12 parameters reflected high correlation; and 2 of 12 showed very high correlations (table 3). ICC2,5 values reflected much higher intrasession reliability: 11 of 12 SCP parameters reflected very high intrasession correlation, and 1 of 12 indicated high correlation (see table 3). Paired t tests showed no significant differences between the first and fifth trial for all parameters.
Table 3Intra- and Intersession ICC, Standard Error of Measurement, and MDC Values for the SCP in AB Individuals
Based on the first trials of each session, intersession ICC2,1 indicated that 6 of 12 SCP parameters had low correlation; 2 of 12 indicated moderate correlation; 3 of 12 had high correlation; and 1 had very high correlation. ICC2,1 values based on the mean of 5 repeated measures increased reliability for most parameters. Six of 12 showed high correlation; 3 of 12 showed moderate correlation; and 3 of 12 showed low correlation.
Standard error of measurement and MDC values at the 95% confidence level are provided in table 3. Similar to WCUs, standard error of measurement and MDC values based on the average of 5 repeated measures were lower for all SCP parameters. Bland-Altman plots revealed no systematic biases between sessions, for all parameters (see fig 1).
Discussion
Previous studies have examined the intra- and intersession reliability of clinical protocols and measurements in gait and manual wheeling.
Test-retest reliability and minimal detectable change determination for 3-dimensional tibial and femoral accelerations during treadmill walking in knee osteoarthritis patients.
Therefore, characterizing the reliability of protocols is well advised. Using similar methodology, to the best of our knowledge this was the first study to examine the reliability of the SCP.
The observed ICC2,1 values in this study show excellent intrasession reliability for all SCP parameters in the WCU group. As expected, averaging 5 trials further increased the intrasession reliability for all parameters. This occurs because averaging the results of multiple trials decreases the influence of the error variance compared with the true score.
Estimating the reliability of continuous measures with Cronbach's alpha or the intraclass correlation coefficient: toward the integration of two traditions.
The significance of using averages of multiple trials is especially illustrated in the intrasession reliability for the AB group. Using ICC2,1, only 8 parameters had values that indicated high or very high intra-session reliability. When using the average of 5 trials (ICC2,5), all SCP parameters showed very high or high intrasession reliability. This highlights the importance of using multiple trials to obtain stable intrasession measures, especially in individuals with no prior wheeling experience (eg, injured individuals getting their first chair).
In both AB individuals and WCUs, paired t tests showed no significant differences between the first and fifth trials for all parameters. This means that a learning effect or fatigue was unlikely within each session.
WCUs showed excellent intersession reliability between the first trials of each session and between the mean of 5 trials, whereas AB individuals showed poor intersession reliability when comparing just the first trials of each session. This was expected because of the large intrasession variability in AB individuals, as indicated by the low intrasession ICC2,1 values when averages of trials were not used. By taking the mean of 5 trials, ICC2,1 values for intersession reliability increased. For AB individuals, however, ICC2,1 based on mean session values for push force and push length on tile, force on ramp, and frequency on carpet still remained low, with unacceptable values that indicated low intersession reliability. As shown by the Bland-Altman plots, no learning effects or systematic differences were observed between sessions for either WCUs or AB individuals when the means of 5 trials were used.
Differences in wheeling variability between AB individuals and WCUs illustrate the importance of manual wheeling experience. The effect of practice has been shown to improve propulsion technique and coordination.
showed that a 12-minute practice session is inadequate for familiarization and developing consistency in timing variables such as push frequency. They observed no improvements in interstroke consistency after a 12-minute practice period or during 3 weeks of practice.
This shows the importance of sufficient wheeling experience in order to decrease variability in wheeling and may partly explain the large variability observed in AB individuals compared with WCUs in the current study.
In general, WCUs showed higher intra- and intersession reliability compared with AB individuals. For WCUs, if there is only time to do 1 trial per surface within a given clinical situation, the ICC values for intra- and intersession reliability showed acceptable values for all parameters of the SCP using only the first trial of each session. Thus, it may be clinically acceptable to conduct only 1 trial per surface for experienced WCUs. However, for AB individuals, averaging the 5 trials of each session is needed to increase both the intra- and intersession reliability of the SCP to a more acceptable level. Therefore, for inexperienced or new WCUs, averaging multiple trials is highly recommended. For the parameters that still indicate poor reliability despite using means, MDC values may provide a more helpful guide for clinical judgments.
ICC is a measure of relative reliability. Therefore, low levels of intersubject variability will lower ICC values even if differences between test sessions are small.
Unlike ICCs, standard error of measurement and MDC are measures of absolute reliability. MDC values could be more useful to clinicians, because they are quantified in the same units as the parameters of interest.
MDC values are especially useful in determining whether changes in values for wheeling parameters are indicative of real change and not measurement error. The MDC values of this study could allow clinicians to estimate with 95% confidence the minimal amount of change in patient status that can be assessed confidently between test sessions. For instance, if a WCU undergoes propulsion training and increases his push length on tile by 12° (as measured by the SCP), based on an estimated MDC of 11.19° (see table 2), we could consider with 95% confidence that the change is because of the intervention and not because of random variation in measurement. This is because the observed change of 12° exceeds the estimated MDC of 11.19°.
In both WCUs and AB individuals, the responsiveness of the SCP is increased (lower MDC) when the mean of 5 trials is used. The AB group showed higher MDC values for all parameters. Therefore, if our AB group were to reflect newly injured individuals who are prescribed with wheelchairs for the first time, the threshold for real change that can be assessed with 95% confidence would be higher than in individuals with manual wheeling experience. Again, this highlights the benefits (especially in individuals with minimal experience) of averaging multiple trials to increase the intersession reliability of the SCP.
Study Limitations
The most significant limitation for this study was the use of AB individuals to represent naïve, newly injured WCUs. This comparison may not be entirely valid because of differences in motor function (ie, trunk control). It would have been difficult to include naïve WCUs with acute injuries, because patients during this stage are in the midst of rehabilitation. However, we and other authors believe that AB individuals naïve to manual wheeling and newly injured WCUs share similar aspects of inexperienced wheeling.
For future directions, the SCP may be modified to include the collection of multiple trials. The optimal number of trials to collect should also be determined. The current SmartWheel clinical software does not include a built-in function to allow for the averaging of multiple trials. Future software updates could address this.
Conclusions
In this study, the test-retest reliability was quantified for each parameter of the SCP. In general, the SCP showed excellent intra- and intersession reliability in individuals with prior wheelchair experience (at least 1y), even when only the first trial is used from a session. However, for individuals with no wheeling experience, collecting and averaging multiple trials is strongly recommended in order to increase the reliability of the SCP. MDC estimates were presented to provide insight into the sensitivity of the SCP in detecting statistically significant changes in wheelchair propulsion.
Preliminary outcomes of the SmartWheel User's Group database: a proposed framework for clinicians to objectively evaluate manual wheelchair propulsion.
Test-retest reliability and minimal detectable change determination for 3-dimensional tibial and femoral accelerations during treadmill walking in knee osteoarthritis patients.
Estimating the reliability of continuous measures with Cronbach's alpha or the intraclass correlation coefficient: toward the integration of two traditions.
Supported by the Natural Sciences and Engineering Research Council (grant no. RGPIN 249489-07 ).
No commercial party having a direct financial interest in the results of the research supporting this article has or will confer a benefit on the authors or on any organization with which the authors are associated.
In-press corrected proof published online on Sep 14, 2012, at www.archives-pmr.org.
00547-3&id=gr1.jpg){kind=link}