Advertisement

Applying the Knowledge-to-Action Framework to Implement Gait and Balance Assessments in Inpatient Stroke Rehabilitation

Open AccessPublished:November 26, 2020DOI:https://doi.org/10.1016/j.apmr.2020.10.133

      Abstract

      Objectives

      The overall objectives of this project were to implement and sustain use of a gait assessment battery (GAB) that included the Berg Balance Scale, 10-meter walk test, and 6-minute walk test during inpatient stroke rehabilitation. The study objective was to assess the effect of the study intervention on clinician adherence to the recommendations and its effect on clinician perceptions and the organization.

      Design

      Pre- and post-training intervention study.

      Setting

      Subacute inpatient rehabilitation facility.

      Participants

      Physical therapists (N=6) and physical therapist assistants (N=2).

      Intervention

      The intervention comprised a bundle of activities, including codeveloping and executing the plan with clinicians and leaders. The multicomponent implementation plan was based on the Knowledge-to-Action Framework and included implementation facilitation, implementation leadership, and a bundle of knowledge translation interventions that targeted barriers. Implementation was an iterative process in which results from one implementation phase informed planning of the next phase.

      Main Outcome Measures

      Clinician administration adherence, surveys of perceptions, and organizational outcomes.

      Results

      Initial adherence to the GAB was 46% and increased to more than 85% after 6 months. These adherence levels remained consistent 48 months after implementation. Clinician perceptions of measure use were initially high (>63%), with significant improvements in knowledge and use of one measure after implementation.

      Conclusions

      We successfully implemented the assessment battery with high levels of adherence to recommendations, likely because of using the bundle of knowledge translation activities, facilitation, and use of a framework to codevelop the plan. These changes in practice were sustainable, as determined by a 4-year follow-up.

      Keywords

      List of abbreviations:

      6MWT (6-minute walk test), 10MWT (10-meter walk test), BBS (Berg Balance Scale), GAB (gait assessment battery), IU (Indiana University), KTA (Knowledge-to-Action cycle), KT (knowledge translation), MFB (Mary Free Bed Rehabilitation Hospital), ORIC (Organizational Readiness to Implement Change), SRALab (Shirley Ryan AbilityLab)
      Measurement is integral to physical rehabilitation, as it can determine a patient’s outcomes, assess the effect of care, increase patient engagement, and determine the value of rehabilitation.
      Guide to physical therapist practice: measurement and outcomes.
      In 2007, the Institute of Medicine recommended the collection and analysis of measures in clinical practice to build a learning health system.
      In this system, measures assess patients’ perspectives and improve care, increase the transparency of outcomes, link clinicians’ performance to patient outcomes and benchmarks, strengthen public health, and generate knowledge.
      The American Physical Therapy Association initiated task forces in 2009 to develop measurement recommendations and, more recently, published a clinical practice guideline on a core set of measures, including gait assessments, to be used at admission, discharge, and, whenever possible, in between these periods.
      • Potter K.
      • Cohen E.T.
      • Allen D.D.
      • et al.
      Outcome measures for individuals with multiple sclerosis: recommendations from the American Physical Therapy Association Neurology Section task force.
      • Sullivan J.E.
      • Crowner B.E.
      • Kluding P.M.
      • et al.
      Outcome measures for individuals with stroke: process and recommendations from the American Physical Therapy Association neurology section task force.
      • Kahn J.H.
      • Tappan R.
      • Newman C.P.
      • et al.
      Outcome measure recommendations from the Spinal Cord Injury EDGE Task Force.
      • McCulloch K.L.
      • de Joya A.L.
      • Hays K.
      • et al.
      Outcome measures for persons with moderate to severe traumatic brain injury: recommendations from the American Physical Therapy Association Academy of Neurologic Physical Therapy TBI EDGE Task Force.
      • Moore J.L.
      • Potter K.
      • Blankshain K.
      • Kaplan S.L.
      • O'Dwyer L.C.
      • Sullivan J.E.
      A core set of outcome measures for adults with neurologic conditions undergoing rehabilitation: a clinical practice guideline.
      Although experts and professional associations recommend routine standardized measurement,
      • Sullivan J.E.
      • Crowner B.E.
      • Kluding P.M.
      • et al.
      Outcome measures for individuals with stroke: process and recommendations from the American Physical Therapy Association neurology section task force.
      ,
      • Moore J.L.
      • Potter K.
      • Blankshain K.
      • Kaplan S.L.
      • O'Dwyer L.C.
      • Sullivan J.E.
      A core set of outcome measures for adults with neurologic conditions undergoing rehabilitation: a clinical practice guideline.
      ,
      • Winstein C.J.
      • Stein J.
      • Arena R.
      • et al.
      Guidelines for adult stroke rehabilitation and recovery: a guideline for healthcare professionals from the American Heart Association/American Stroke Association.
      research demonstrates that clinicians rely on experience to guide decision-making.
      • Jette D.U.
      • Halbert J.
      • Iverson C.
      • Miceli E.
      • Shah P.
      Use of standardized outcome measures in physical therapist practice: perceptions and applications.
      • Swinkels R.A.
      • van Peppen R.P.
      • Wittink H.
      • Custers J.W.
      • Beurskens A.J.
      Current use and barriers and facilitators for implementation of standardised measures in physical therapy in the Netherlands.
      • Colquhoun H.L.
      • Lamontagne M.E.
      • Duncan E.A.
      • Fiander M.
      • Champagne C.
      • Grimshaw J.M.
      A systematic review of interventions to increase the use of standardized outcome measures by rehabilitation professionals.
      Barriers and facilitators to using standardized measures exist in health care and may be categorized at individual and organizational levels.
      • Duncan E.A.
      • Murray J.
      The barriers and facilitators to routine outcome measurement by allied health professionals in practice: a systematic review.
      Individual clinician-level barriers and facilitators include the perceived value of measurement, as well as the knowledge, skill, and educational level of the practitioner.
      • Duncan E.A.
      • Murray J.
      The barriers and facilitators to routine outcome measurement by allied health professionals in practice: a systematic review.
      Clinicians’ perceptions of patient-related barriers include concerns of sharing measurement information with patients, questioning the relevance of measures, and the potential for a negative effect if measures fail to show progress.
      • Duncan E.A.
      • Murray J.
      The barriers and facilitators to routine outcome measurement by allied health professionals in practice: a systematic review.
      Organizational barriers include low support or level of priority for the use of measures and poor cooperation of colleagues and managers.
      • Duncan E.A.
      • Murray J.
      The barriers and facilitators to routine outcome measurement by allied health professionals in practice: a systematic review.
      Other practical considerations include the time required to learn and administer the measures and the availability of equipment.
      • Duncan E.A.
      • Murray J.
      The barriers and facilitators to routine outcome measurement by allied health professionals in practice: a systematic review.
      The field of knowledge translation (KT) aims to identify efficient and effective methods to implement evidence into clinical practice. KT is the dynamic and iterative process that includes synthesis, dissemination, exchange, and ethically-sound application of knowledge to improve health, provide more effective health services and products, and strengthen the health care system.
      Canadian Institutes of Health Research.
      A recent systematic review assessed the types and effectiveness of KT interventions (ie, strategies to change behavior and promote adoption) to increase knowledge of, improve attitudes toward, and enhance the use of standardized outcome measures in the field of rehabilitation.
      • Colquhoun H.L.
      • Lamontagne M.E.
      • Duncan E.A.
      • Fiander M.
      • Champagne C.
      • Grimshaw J.M.
      A systematic review of interventions to increase the use of standardized outcome measures by rehabilitation professionals.
      Of 11 studies, 9 used educational strategies (eg, educational workshops or seminars) and 2 used indirect strategies (eg, providing resources and written guidelines) to disseminate measurement information. Five studies combined 3 or more strategies, 3 studies applied audit and feedback, and 1 study used knowledge brokers (ie, individuals to guide and support KT processes).
      • Colquhoun H.L.
      • Lamontagne M.E.
      • Duncan E.A.
      • Fiander M.
      • Champagne C.
      • Grimshaw J.M.
      A systematic review of interventions to increase the use of standardized outcome measures by rehabilitation professionals.
      Nine of the 10 studies that assessed rates of measurement use reported improvements in perceived (n=5) or actual (n=4) use. KT theory informed only 5 studies in this review, and all studies targeted individual clinicians (vs organizations). Two studies reported on sustainability.
      • Colquhoun H.L.
      • Lamontagne M.E.
      • Duncan E.A.
      • Fiander M.
      • Champagne C.
      • Grimshaw J.M.
      A systematic review of interventions to increase the use of standardized outcome measures by rehabilitation professionals.
      Russell et al
      • Russell D.J.
      • Rivard L.M.
      • Walter S.D.
      • et al.
      Using knowledge brokers to facilitate the uptake of pediatric measurement tools into clinical practice: a before-after intervention study.
      reported that effects were sustained for 3 of 4 implemented measures after 12 and 18 months based on clinician report of “high” amounts by approximately 60% to 85% of survey respondents. Schreiber et al
      • Schreiber J.
      • Marchetti G.F.
      • Racicot B.
      • Kaminski E.
      The use of a knowledge translation program to increase use of standardized outcome measures in an outpatient pediatric physical therapy clinic: administrative case report.
      documented an increase in the frequency of 5 of 6 implemented measures and sustained over 8 months. However, the authors did not provide the percentage of patients assessed with these measures.
      • Schreiber J.
      • Marchetti G.F.
      • Racicot B.
      • Kaminski E.
      The use of a knowledge translation program to increase use of standardized outcome measures in an outpatient pediatric physical therapy clinic: administrative case report.
      Use of systematic processes and implementation frameworks may improve the use of research by clinicians.
      • Fischer F.
      • Lange K.
      • Klose K.
      • Greiner W.
      • Kraemer A.
      Barriers and strategies in guideline implementation-a scoping review.
      The Knowledge-to-Action Cycle (KTA), a planned-action framework, is the product of a metasynthesis of more than 30 planned action models. This framework is composed of 7 phases to implement evidence into clinical practice and includes the knowledge creation funnel at the center, which represents the publication, synthesis of research, and knowledge products and tools such as practice guidelines.
      • Graham I.D.
      • Logan J.
      • Harrison M.B.
      • et al.
      Lost in knowledge translation: time for a map?.
      ,
      • Straus S.
      • Tetroe J.
      • Graham I.
      Knowledge Translation in health care: moving from evidence to practice.
      The overarching goal of this KT project was to implement an evidence-based gait assessment battery (GAB) with high levels of adherence in inpatient stroke rehabilitation and determine the sustainability of this new practice. The intervention included codeveloping and executing an implementation plan with clinicians and leaders. The multicomponent implementation plan was based on the KTA Framework and included implementation facilitation, implementation leadership, and a bundle of KT interventions that targeted barriers. Implementation was an iterative process in which results from one implementation phase informed planning of the next phase. The study objective was to assess the effect of the study intervention on clinician adherence to the recommendations and its effect on clinician perceptions and the organization.

      Methods

      We implemented the GAB at Mary Free Bed Rehabilitation Hospital (MFB) in Grand Rapids, Michigan. This nonprofit hospital has 119 acute rehabilitation and 48 skilled nursing rehabilitation beds. In 2015, there were 342 stroke admissions, 6 physical therapists (4.0 full-time positions), and 2 physical therapist assistants (1.0 full-time position) on the stroke unit. Clinicians reported using measures at their discretion before implementation, but measures varied among clinicians.
      A primary goal of this project was to implement a GAB into routine clinical practice on a stroke rehabilitation unit. However, patients provided informed consent for research use of the GAB results. Inclusion criteria were adult patients (aged <90y) with subacute stroke (<2mo poststroke) and goals to improve walking. If patients were unable to provide informed consent, a legally authorized representative could provide it. Exclusion criteria included an inability to ambulate more than 50 meters before the stroke. Clinicians provided informed consent before taking online surveys. Inclusion criteria for clinicians included full- or part-time physical therapist or physical therapist assistant on the MFB stroke unit. The Northwestern University, Indiana University (IU), and MFB Institutional Review Boards approved the project.
      The KTA Framework guided the development of the KT plan, and the methods are described according to the KTA action cycle phases (see subheadings). Integrated KT was a core principle of the KT plan, which is “a model of collaborative research, where researchers work with knowledge users who identify a problem and have the authority to implement the research recommendations.”
      • Kothari A.
      • McCutcheon C.
      • Graham I.D.
      Defining integrated knowledge translation and moving forward: a response to recent commentaries.
      (p299) The research, leadership, and clinical teams worked collaboratively on the development and implementation of the KT plan. The team also considered issues that might affect sustainability of the new practice at the beginning of and throughout the project. When possible, the team integrated the KT interventions into standard organizational processes (ie, team conference reporting, performance goals). The team used an iterative approach to implement the plan and repeated phases until the achievement of desired results occurred. In addition, results from early phases informed the activities of later phases. Therefore, the KTA phase activities and associated results are presented here in the methods (table 1). The results section presents the implementation outcomes.
      Table 1Implementation plan and results with activities described according to each phase of the KTA cycle
      KTA PhaseMethods for Each PhaseResults
      Identify problem; determine the know-do gap; identify, review, and select knowledge
      • 1.
        Adapted survey
        • Swinkels R.A.
        • van Peppen R.P.
        • Wittink H.
        • Custers J.W.
        • Beurskens A.J.
        Current use and barriers and facilitators for implementation of standardised measures in physical therapy in the Netherlands.
        on perceptions, barriers, and facilitators related to standardized measures and informal discussions
      • 2.
        Selected knowledge (ie, gait and balance assessments) for implementation
      • 1.
        Assessment of know-do gap: positive perception of measures, several measures used at the therapists’ discretion, but none were required or standardized
      • 2.
        Knowledge selected: BBS, 10MWT, 6MWT
      Adapt knowledge to local context
      • 1.
        Review of current evidence for GAB in subacute patient populations
      • 2.
        Adaptation of the standardized administration procedures to fit into local context; recommendations for adaptations made by clinicians, administrators, and researchers
      Recommendations for:
      • 1.
        Use of GAB in subacute stroke to support interpretation of results
      • 2.
        Administration procedures: a. Administration protocol adapted to the MFB context
        • b.
          Administration timing (within 72 h of admission, weekly, and at discharge)
        • c.
          Documentation recommendations
      Assess barriers and facilitators to knowledge use
      • 1.
        Survey to MFB clinicians that included adapted survey
        • Swinkels R.A.
        • van Peppen R.P.
        • Wittink H.
        • Custers J.W.
        • Beurskens A.J.
        Current use and barriers and facilitators for implementation of standardised measures in physical therapy in the Netherlands.
        on perceptions, barriers, and facilitators and the Organizational Readiness to Implement Change survey
        • Shea C.M.
        • Jacobs S.R.
        • Esserman D.A.
        • Bruce K.
        • Weiner B.J.
        Organizational readiness for implementing change: a psychometric assessment of a new measure.
      • 2.
        Informal discussions about barriers and facilitators
      • 3.
        An iterative process of barrier and facilitator assessment, implementation of KT interventions, and monitoring occurred for 6 months until adherence consistently achieved >85%
      • 1.
        Barriers included knowledge, skills, social influences, and environmental context and resources (see table 3)
      Select, tailor, implement interventions
      • 1.
        Barriers were categorized according to the Theoretical Domains Framework and KT interventions were selected
        • Atkins L.
        • Francis J.
        • Islam R.
        • et al.
        A guide to using the Theoretical Domains Framework of behaviour change to investigate implementation problems.
        ,
        • Michie S.
        • Johnston M.
        • Francis J.
        • Hardeman W.
        • Eccles M.
        From theory to intervention: mapping theoretically derived behavioural determinants to behaviour change techniques.
      • 2.
        Design of KT interventions codeveloped with the MFB clinicians and research team
      • 1.
        A multi-component KT intervention was delivered that included educational interventions, leadership support, process changes, audit and feedback, purchasing of equipment, and environmental modifications (see table 3)
      Monitor knowledge use
      • 1.
        Monitored conceptual use with previously administered surveys
      • 2.
        Monitored instrumental use with audits and during team conference reporting
      • 1.
        Survey results demonstrated statistically significant increase in use of the 10MWT
      • 2.
        >85% adherence achieved and sustained after 6 mo
      Evaluate outcomes
      • 1.
        Clinician outcomes assessed using previously administered surveys
      • 2.
        Organizational outcomes assessed anecdotally
      • 1.
        Survey results demonstrated statistically significant difference identified in clinicians’ knowledge and skill related to administration of measures
      • 2.
        Observations reported by Butzer et al
        • Butzer J.F.
        • Virva R.
        • Lenca L.
        Commentary on the challenges and benefits of implementing standardized outcome measures.
      Sustain knowledge use
      • 1.
        Process changes: ongoing monitoring of adherence, modification of electronic health record, weekly reporting in team conference
      • 2.
        Staff: adopted standardized testing day, standardization processes integrated into new hire orientation, practice incorporated into performance appraisals and job description
      • 3.
        Organization: incorporated into the goals/vision, implemented throughout system of care
      • 1.
        All KT interventions phased out, with the exception of scanning of assessment form into medical record.
      • 2.
        >85% adherence continued until last audit at 24 mo
      • 3.
        100% of clinicians agreed or strongly agreed on the survey that the project increase value of GAB and the culture is data-driven
      NOTE. The activities describe the methods used for each phase, and the results column provides an overview of the findings from each KTA phase.

      Identify the problem, assess know-do gap, select knowledge to implement

      The research and clinical team discussed appropriate measurement areas and the average impairment level of the patients at admission to identify an appropriate GAB. An important consideration in measurement selection was the team’s future plans to implement a high-intensity gait training program. Therefore, the measures should target areas of function that improve with this intervention. The clinical team reported that patients often required moderate to total assistance (ie, 50%-100% assistance) for mobility and balance when admitted to inpatient rehabilitation. Therefore, we aimed to select measures that assess static and dynamic sitting and standing balance, independence with walking, gait speed, and walking distance. Measurement selection was based on the measures’ potential to demonstrate change in patients with the typical level of function at admission. In addition, the measure needed excellent psychometrics and clinical utility. The research team evaluated recommendations made by the American Physical Therapy Association StrokeEDGE group
      • Sullivan J.E.
      • Crowner B.E.
      • Kluding P.M.
      • et al.
      Outcome measures for individuals with stroke: process and recommendations from the American Physical Therapy Association neurology section task force.
      and published the 2016 Stroke Rehabilitation Clinical Practice Guideline
      • Winstein C.J.
      • Stein J.
      • Arena R.
      • et al.
      Guidelines for adult stroke rehabilitation and recovery: a guideline for healthcare professionals from the American Heart Association/American Stroke Association.
      to identify appropriate measures for the GAB. The GAB recommendations included the 10-meter walk test (10MWT), including assistance levels;
      • Sullivan J.E.
      • Crowner B.E.
      • Kluding P.M.
      • et al.
      Outcome measures for individuals with stroke: process and recommendations from the American Physical Therapy Association neurology section task force.
      ,
      • Winstein C.J.
      • Stein J.
      • Arena R.
      • et al.
      Guidelines for adult stroke rehabilitation and recovery: a guideline for healthcare professionals from the American Heart Association/American Stroke Association.
      6-minute walk test (6MWT), including assistance levels
      • Sullivan J.E.
      • Crowner B.E.
      • Kluding P.M.
      • et al.
      Outcome measures for individuals with stroke: process and recommendations from the American Physical Therapy Association neurology section task force.
      ,
      • Winstein C.J.
      • Stein J.
      • Arena R.
      • et al.
      Guidelines for adult stroke rehabilitation and recovery: a guideline for healthcare professionals from the American Heart Association/American Stroke Association.
      ; and the Berg Balance Scale (BBS).
      • Sullivan J.E.
      • Crowner B.E.
      • Kluding P.M.
      • et al.
      Outcome measures for individuals with stroke: process and recommendations from the American Physical Therapy Association neurology section task force.
      ,
      • Winstein C.J.
      • Stein J.
      • Arena R.
      • et al.
      Guidelines for adult stroke rehabilitation and recovery: a guideline for healthcare professionals from the American Heart Association/American Stroke Association.
      ,
      • Blum L.
      • Korner-Bitensky N.
      Usefulness of the Berg Balance Scale in stroke rehabilitation: a systematic review.
      Two years later, the Core Set of Outcome Measures for Neurologic Clinical Practice also recommended these measures for clinical practice, which further supports the measures selected for implementation.
      • Moore J.L.
      • Potter K.
      • Blankshain K.
      • Kaplan S.L.
      • O'Dwyer L.C.
      • Sullivan J.E.
      A core set of outcome measures for adults with neurologic conditions undergoing rehabilitation: a clinical practice guideline.
      Once selected, the research team reviewed the GAB with the clinical and leadership team and did not recommend changes. To formally assess perceptions, barriers, and current use of outcome measures in clinical practice (ie, the know-do gap), we used a previously validated survey
      • Swinkels R.A.
      • van Peppen R.P.
      • Wittink H.
      • Custers J.W.
      • Beurskens A.J.
      Current use and barriers and facilitators for implementation of standardised measures in physical therapy in the Netherlands.
      with additional questions about the selected measures (table 2). We administered the online surveysa before implementation (2015), and 1 and 3 years after implementation.
      Table 2Investigator developed survey questions
      Questions Added to the 2015, 2016, and 2018 Survey, With the 2015 Results Described
      Please read the following questions on the effect of the FIRST project on your measurement related practice. Indicate whether you completely disagree (1), disagree (2), are neutral (3, neither agree nor disagree), agree (4), or completely agree (5).
       1. I am familiar with the Berg Balance Scale4.5 (4.0-5.0)
       2. I use the Berg Balance Scale in routine practice5.0 (4.0-5.0)
       3. I am familiar with the 10-meter walk test4.0 (3.25-4.0)
       4. I use the 10-meter walk test as part of routine practice3.0 (2.0-4.0)
       5. I am familiar with the 6-minute walk test4.0 (4.0-5.0)
       6. I use the 6-minute walk test as part of routine practice4.0 (4.0-4.75)
      Questions Added to the 2018 Survey, With 2018 Results Described
      Please read the following questions on the effect of the project on your measurement related practice. Indicate whether you completely disagree (1), disagree (2), are neutral (3, neither agree nor disagree), agree (4), or completely agree (5).
       1. As a result of this project, I have increased the use of outcome measures in my clinical practice.5.0 (4.25-5.0)
       2. As a result of this project, I use the outcome measure results to guide my clinical decision-making.5.0 (4.0-5.0)
       3. As a result of this project, I have more discussions with my patients about their outcome measurement results.5.0 (4.0-5.0)
       4. As a result of this project, I have more discussions with colleagues about outcome measurement results.4.5 (4.0-5.0)
       5. As a result of this project, the culture in our department has shifted to discuss patient-related data (ie, outcome measurement results) instead of patient observations (ie, patient walks slowly, has poor balance, etc).4.0 (4.0-5.0)
       6. As a result of this project, I better understand the value outcome measures add to clinical practice.5.0 (4.0-5.0)
       7. After this project is over, I plan to use outcome measures at the same or higher frequency that I currently use them.4.0 (4.0-5.0)
      NOTE. Results provided in median (range).

      Adapt the knowledge about the GAB to the local context

      The research and clinical team reviewed and adapted standardization procedures for application in the MFB context. Adaptations included recommendations for GAB administration within 72 hours of admission, weekly, and at discharge. Measurement-specific adaptations included standing on the weaker limb during the BBS single-limb support and tandem items,
      • Straube D.
      • Moore J.
      • Leech K.
      • Hornby T.G.
      Item analysis of the berg balance scale in individuals with subacute and chronic stroke.
      and allowing physical assistance during the 10MWT
      • Fulk G.D.
      • Echternach J.L.
      Test-retest reliability and minimal detectable change of gait speed in individuals undergoing rehabilitation after stroke.
      and 6MWT
      • Fulk G.D.
      • Echternach J.L.
      • Nof L.
      • O'Sullivan S.
      Clinometric properties of the six-minute walk test in individuals undergoing rehabilitation poststroke.
      because of the high number of patients who require assistance during inpatient rehabilitation.
      • Hornby T.G.
      • Holleran C.L.
      • Leddy A.L.
      • et al.
      Feasibility of focused stepping practice during inpatient rehabilitation poststroke and potential contributions to mobility outcomes.

      Assess barriers and facilitators to implementation of the GAB

      The research team administered an outcome measurement barrier survey while assessing current practice.
      • Swinkels R.A.
      • van Peppen R.P.
      • Wittink H.
      • Custers J.W.
      • Beurskens A.J.
      Current use and barriers and facilitators for implementation of standardised measures in physical therapy in the Netherlands.
      The Organizational Readiness to Implement Change (ORIC) questionnaire assessed the clinicians’ perception of the organization’s readiness for this change.
      • Shea C.M.
      • Jacobs S.R.
      • Esserman D.A.
      • Bruce K.
      • Weiner B.J.
      Organizational readiness for implementing change: a psychometric assessment of a new measure.
      Areas assessed included the staff’s confidence, motivation, and determination to implement the GAB. The ORIC identified relevant barriers and organizational readiness for change, and guided the selection of organizational level KT interventions.
      • Shea C.M.
      • Jacobs S.R.
      • Esserman D.A.
      • Bruce K.
      • Weiner B.J.
      Organizational readiness for implementing change: a psychometric assessment of a new measure.
      Informal barrier assessments during meetings and daily interactions with clinicians occurred until adherence levels reached more than 85%. The research team categorized the barriers and facilitators using the theoretical domains framework, which is a comprehensive, theory-informed approach to identify determinants of behavior.
      • Atkins L.
      • Francis J.
      • Islam R.
      • et al.
      A guide to using the Theoretical Domains Framework of behaviour change to investigate implementation problems.
      ,
      • Michie S.
      • Johnston M.
      • Francis J.
      • Hardeman W.
      • Eccles M.
      From theory to intervention: mapping theoretically derived behavioural determinants to behaviour change techniques.
      Barriers identified by clinicians included a lack of knowledge and skills to administer the tests and incorporate results into clinical decision-making. Clinicians also reported conflicts between the recommended practice and current beliefs. Some examples are the belief that prioritizing assessments would decrease the time available for treatment and that assessment should not occur on patients who might perform poorly (eg, BBS ≤5). Other barriers included the environment and resources, such as the lack of equipment and space. One clinician stated, “No [barriers]….just need the motivation.” The identified categories of barriers were knowledge, skills, social influences, and environmental context and resources (table 3).
      • Atkins L.
      • Francis J.
      • Islam R.
      • et al.
      A guide to using the Theoretical Domains Framework of behaviour change to investigate implementation problems.
      ,
      • Michie S.
      • Johnston M.
      • Francis J.
      • Hardeman W.
      • Eccles M.
      From theory to intervention: mapping theoretically derived behavioural determinants to behaviour change techniques.
      Table 3Barrier categories and knowledge translation interventions.
      BarrierKT Intervention
      Knowledge and skillsEducation sessions: training and standardization of GAB

      Consultations, structured meeting, and informal discussion

      Education session on interpretation of results

      Cheat sheet developed for SEMs, MDCs, goal writing

      Annual standardization
      Social influencesLeadership support (articulation of clear expectations and reinforcement)

      Required reporting at team conference

      Specific day for all PTs to perform outcome measures

      Audit and feedback

      Rewards
      Environmental context and resourcesPurchase equipment (measuring wheels, 10-meter rope, stopwatch, etc)

      Research assistant support for data collection, timing, putting equipment out for easy access, and reminders

      Development of data collection forms, scanned into EMR
      Abbreviation: EMR, electronic medical record.
      Facilitators included organizational, social, individual, and financial factors. MFB’s vision included the integration of outcome measures and the creation of a learning health system, which aligned with the goals of this project and could serve as an organizational KT strategy. Key stakeholders, including the leaders, clinicians, and MFB researchers, were also engaged and actively involved in the project. Lastly, funding supported personnel, training, and equipment.

      Select and tailor KT interventions to implement the GAB

      The research team and clinicians codeveloped the multicomponent KT plan that targeted the identified barriers. Using the Theoretical Domains Framework
      • Atkins L.
      • Francis J.
      • Islam R.
      • et al.
      A guide to using the Theoretical Domains Framework of behaviour change to investigate implementation problems.
      ,
      • Michie S.
      • Johnston M.
      • Francis J.
      • Hardeman W.
      • Eccles M.
      From theory to intervention: mapping theoretically derived behavioural determinants to behaviour change techniques.
      to map KT interventions to barriers, we selected interventions that were deemed feasible and suitable in the MFB context. To target barriers of knowledge and skills, we held an educational session on measurement concepts, psychometric properties, and clinical utility of the GAB, and standardization of its administration (see appendix 1 for details). Clinicians completed a post-test by rating patients on a video demonstration of the assessments. Clinician scores averaged 96% initially and 89% at 1 year later. Items that had lower levels of correct responses were clarified. The research-clinician team developed a resource sheet to assist in test interpretation and inclusion of the GAB in goal writing. Clinicians also received training on discussing GAB results with patients.
      The KT interventions that targeted social influences included leadership support, team conference reporting, audit and feedback,
      • Colquhoun H.L.
      • Carroll K.
      • Eva K.W.
      • et al.
      Advancing the literature on designing audit and feedback interventions: identifying theory-informed hypotheses.
      ,
      • Colquhoun H.
      • Michie S.
      • Sales A.
      • et al.
      Reporting and design elements of audit and feedback interventions: a secondary review.
      and the determination of an administration day. The Research Director, Stroke Manager, and research team collaboratively set an adherence goal of administration of greater than 85% of expected assessments, and clinicians reported that the goal was feasible. The leadership team required reporting of GAB results at the weekly team conference. They also reinforced these expectations often and provided rewards (eg, catered lunches, baked goods) for the achievement of desired adherence levels. The research team audited charts and provided feedback on adherence every 4 to 6 weeks. This process consisted of data extraction from each patient’s chart (ie, assessment collected: yes or no), calculation of percentage administered (number of yes and total number of expected assessments), and reporting back to the team by e-mail or team meeting. E-mails were sent to the clinicians by the MFB research team with a copy to the unit manager. Adherence rates of less than 85% triggered additional barrier assessments and KT interventions. A full description of the audit and feedback intervention
      • Colquhoun H.
      • Michie S.
      • Sales A.
      • et al.
      Reporting and design elements of audit and feedback interventions: a secondary review.
      is available in appendix 2. Lastly, adaptations to the administration timing recommendations occurred to implement a standard weekly interim testing day (nicknamed “Testing Tuesday”).
      The research-clinician team secured several environmental resources. Equipment, including measuring wheels and sticks, were placed in the gym. MFB staff painted lines to mark distances for the 10MWT on baseboards. The research assistant reminded clinicians about testing and assisted in data collection, timing, and setting up equipment. The research-clinical team codeveloped data collection forms that were scanned into the medical record to minimize documentation burden.
      Motivation was identified as a barrier and targeted by KT interventions that also targeted other barriers. Specifically, we stated a goal of greater than 85% adherence and provided information about the outcome of the behavior change interventions (ie, audit and feedback). We also provided rewards and used social processes, such as team conferences, to increase motivation for behavior change.
      During the first 6 months, we used an iterative process that included barrier assessment, codevelopment of KT interventions, and monitoring until adherence was consistently greater than 85%. We stopped providing feedback after 14 months but audited for 24 months. When adherence consistently achieved greater than 85%, the research team phased out the KT interventions. Appendix 1 describes the KT interventions according to Workgroup for Intervention Development and Evaluation Research criteria.
      • Albrecht L.
      • Archibald M.
      • Arseneau D.
      • Scott S.D.
      Development of a checklist to assess the quality of reporting of knowledge translation interventions using the Workgroup for Intervention Development and Evaluation Research (WIDER) recommendations.

      Monitor knowledge use of the GAB and assess outcomes (Effect)

      The research team monitored the GAB throughout the 24-month project. We assessed conceptual knowledge use by readministering the initial surveys on perceptions, barriers, and use of outcome measures before and at 1 and 3 years after implementation. The research team monitored instrumental or actual use (adherence to GAB) by conducting chart audits and sharing the adherence rates with the leadership and clinical teams. The ORIC questionnaire assessed the effect of the project on the organization’s confidence, motivation, and ability to implement the change. The leadership team also performed a nonsystematic observation of organization impact.
      • Butzer J.F.
      • Virva R.
      • Lenca L.
      Commentary on the challenges and benefits of implementing standardized outcome measures.

      Sustain use

      The research-clinician team integrated the GAB into routine clinical and organizational processes using a number of sustainability interventions. Clinicians routinely report GAB results and interpretation during team conferences, and training on the GAB is a core component of new hire orientation. The electronic medical record incorporated GAB data fields. The leadership team integrated the GAB into organizational goals and vision. Managers wrote adherence performance appraisal goals and added the GAB to job descriptions. A follow-up chart audit assessed sustainability at 48 months after implementation.

      Statistical analysis

      Descriptive statistics were used to described clinician demographics. Adherence data were collected, and a monthly average was generated for 24 months and at a 48-month follow-up. Friedman’s test determined differences in baseline (2015), early-implementation (2016), and late implementation (2018) survey data. For statistically significant results, pairwise comparisons were performed using SPSS Statistics, version 23.b

      Results

      Clinician demographics are described in table 4. During 24 months, the research team approached 197 eligible patients and 157 patients enrolled (80% enrollment rate). Data were collected over 24 months, and we conducted a 48-month follow-up chart audit on 45 patients.
      Table 4Clinician demographics (n=8)
      Demographicsn (%)
      Age, y
       20-293 (38)
       30-392 (25)
       40-491 (13)
       50-592 (25)
      Sex
       Men2 (25)
       Women6 (75)
      Clinical role
       Physical therapist6 (75)
       Physical therapist assistant2 (25)
      Years of practice
       <54 (50)
       5-100 (0)
       11-151 (12.5)
       >153 (37.5)
      Entry-level professional degree
       Associates2 (25)
       Baccalaureate or equivalent1 (12.5)
       Master’s2 (25)
       PhD0 (0)
       Doctorate2 (25)
       Other advanced degree1 (12.5)
      Clinical specialist (ie, NCS)
       Yes0 (0)
       No8 (100)
      Hours worked per week
       ≤201 (12.5)
       21-302 (25)
       31-391 (12.5)
       ≥404 (50)
      Number of patients seen daily
       3-40 (0)
       5-65 (71.43)
       7-80 (0)
       >82 (28.7)
      Abbreviation: NCS, Board Certified in Neurologic Physical Therapy.

      Knowledge use (GAB adherence)

      At the first chart audit after starting to use KT interventions, adherence to recommendations was 46%. Adherence increased to greater than 85% by 6 months after the initiation of implementation interventions (fig 1). The 48-month follow-up chart audit indicated that adherence levels were at 95.2%.
      Figure thumbnail gr1
      Fig 1Adherence to administration recommendations.

      Clinician perceptions

      Each clinician on the stroke unit (n=8) participated in the survey in 2015, 2016, and 2018. In 2015, 63% of participants agreed or strongly agreed that they used measures for diagnosis, 88% used measures for prognosis, and 100% monitored change over time. However, 38% of clinicians reported measuring 40% of patients, 13% reported measuring 60% of patients, 38% reported measuring 80% of patients, and only 13% reported measuring 100% of patients. Over the study data collection periods (2015, 2016, and 2018), clinicians' knowledge of how to administer the measures significantly improved (2015: median, 4.0; range, 4.0-5.0; 2016: median, 4.0; range, 4.0-5.0; 2018: median, 5.0; range, 5.0-5.0; χ2(2)=8.82; P=.01). Post hoc testing indicated differences between 2015 and 2018 (P=.02). Use of the 10MWT also significantly improved between 2015, 2016, and 2018 (2015: median, 3.0; range, 2.0-4.0; 2016: median, 5.0; range, 4.0-5.0; 2018: median, 5.0; range, 4.25-5.0; χ2(2)=10.33; P=.006). Post hoc testing indicated differences between 2015 and 2018 (P=.03).

      Organizational outcomes

      The ORIC score (median, range) increased from 2015 to 2018 (2015: 52 (39-53.5); 2016: 53.5 (45.5-59.0); 2018: 59.5 (47.5-60.0), but they were not significantly different from each other (P>.05) at the 3 data collection points. The MFB leadership team observed the effect of the project on the organization, including a positive effect on patients and families, demonstration of the value of outcome measurement, and facilitation of achievement of an organizational goal of routine clinical outcome measurement.

      Discussion

      In this project, we codeveloped a multicomponent implementation plan that was based on the KTA Framework and included implementation facilitation, implementation leadership, and a bundle of KT interventions that targeted barriers. Six months after implementation of the GAB, greater than 85% adherence was achieved and sustained for 48 months. Clinician perceptions of measurement improved during the study, and perceptions of barriers decreased. The project positively affected the organization, as illustrated by actions taken by MFB to implement the GAB throughout the organization and through observations described by Butzer et al.
      • Butzer J.F.
      • Virva R.
      • Lenca L.
      Commentary on the challenges and benefits of implementing standardized outcome measures.
      The KT interventions were designed using theory, targeted barriers, and engaged end-users, all of which are considered essential components of designing effective behavior change interventions.
      • Colquhoun H.L.
      • Squires J.E.
      • Kolehmainen N.
      • Fraser C.
      • Grimshaw J.M.
      Methods for designing interventions to change healthcare professionals' behaviour: a systematic review.
      Studies on implementation of measurements often use educational approaches.
      • Colquhoun H.L.
      • Lamontagne M.E.
      • Duncan E.A.
      • Fiander M.
      • Champagne C.
      • Grimshaw J.M.
      A systematic review of interventions to increase the use of standardized outcome measures by rehabilitation professionals.
      These approaches target knowledge barriers and may result in improved knowledge and skill, but do not have a substantial effect on clinical practice.
      • Jones C.A.
      • Roop S.C.
      • Pohar S.L.
      • Albrecht L.
      • Scott S.D.
      Translating knowledge in rehabilitation: systematic review.
      Although our KT interventions included education-based strategies, they constituted a small percentage of the approach. Audit and feedback was frequently used, which alone results in modest improvements in clinical practice.
      • Ivers N.
      • Jamtvedt G.
      • Flottorp S.
      • et al.
      Audit and feedback: effects on professional practice and healthcare outcomes.
      Although the critical elements of audit and feedback are unknown, the intervention included components that may increase its effectiveness, such as feedback by a supervisor or respected colleague, frequent feedback, specific goals, and action-plans (≥85% adherence and codevelopment of KT interventions to target reported barriers).
      • Ivers N.M.
      • Grimshaw J.M.
      • Jamtvedt G.
      • et al.
      Growing literature, stagnant science? Systematic review, meta-regression and cumulative analysis of audit and feedback interventions in health care.
      The integrated KT (ie, active involvement of stakeholders)
      • Gagliardi A.R.
      • Berta W.
      • Kothari A.
      • Boyko J.
      • Urquhart R.
      Integrated knowledge translation (IKT) in health care: a scoping review.
      ,
      • Nguyen T.
      • Graham I.D.
      • Mrklas K.J.
      • et al.
      How does integrated knowledge translation (IKT) compare to other collaborative research approaches to generating and translating knowledge? Learning from experts in the field.
      approach taken, which included ongoing codevelopment of KT interventions to target barriers and selection and tailoring of implementation and sustainability strategies, may have positively affected outcomes.
      Systematic reviews of KT interventions report frequent use of interventions that target individuals.
      • Colquhoun H.L.
      • Lamontagne M.E.
      • Duncan E.A.
      • Fiander M.
      • Champagne C.
      • Grimshaw J.M.
      A systematic review of interventions to increase the use of standardized outcome measures by rehabilitation professionals.
      ,
      • Colquhoun H.L.
      • Squires J.E.
      • Kolehmainen N.
      • Fraser C.
      • Grimshaw J.M.
      Methods for designing interventions to change healthcare professionals' behaviour: a systematic review.
      The KT interventions used in this project also targeted organizational leadership and other environmental and resource barriers. The leadership team emphasized GAB use and articulated its importance to the organization’s vision, which may have facilitated this project’s success.
      • Gifford W.A.
      • Davies B.
      • Edwards N.
      • Graham I.D.
      Leadership strategies to influence the use of clinical practice guidelines.
      • Harlos K.
      • Tetroe J.
      • Graham I.D.
      • Bird M.
      • Robinson N.
      Mining the management literature for insights into implementing evidence-based change in healthcare.
      • Salbach N.M.
      • Jaglal S.B.
      • Korner-Bitensky N.
      • Rappolt S.
      • Davis D.
      Practitioner and organizational barriers to evidence-based practice of physical therapists for people with stroke.
      Clinicians perceived that the organization was ready for this change as indicated by a relatively high aggregate ORIC score. Implementation research funding provided the financial support for MFB staff time, external guidance, and equipment, as well as legitimation to the project goal. Staff had time to prepare testing packets, complete medical record modifications, and prepare the environment for testing (eg, develop BBS test kits, create rope with 10-meter marks on it, etc). It is important to note that this external funding may initiate projects and generate implementation knowledge. However, relying on implementation research funding is not sustainable for long-term or widespread practice changes. Without this implementation research funding at MFB, it is unlikely that the implementation of the GAB would have achieved a high level of adherence. Organizations may benefit from budgeting for KT activities to ensure that adequate resources are available to support successful implementation efforts. Future research should also assess the funding needs for successful implementation.
      Facilitation is a social process that focuses on evidence-informed practice change and includes project management, leadership, relationship building, and communication.
      • Berta W.
      • Cranley L.
      • Dearing J.W.
      • Dogherty E.J.
      • Squires J.E.
      • Estabrooks C.A.
      Why (we think) facilitation works: insights from organizational learning theory.
      External facilitation
      • Berta W.
      • Cranley L.
      • Dearing J.W.
      • Dogherty E.J.
      • Squires J.E.
      • Estabrooks C.A.
      Why (we think) facilitation works: insights from organizational learning theory.
      ,
      • Harvey G.
      • Kitson A.
      Implementing evidence-based practice in healthcare. a facilitation guide.
      guided the implementation process. Open communication between the MFB and the IU and Shirley Ryan AbilityLab (SRALab) team positively affected the success of the project. The MFB research team tailored information to the local context, collaborated to remove barriers, monitored progress, and maintained effective communication.
      • Berta W.
      • Cranley L.
      • Dearing J.W.
      • Dogherty E.J.
      • Squires J.E.
      • Estabrooks C.A.
      Why (we think) facilitation works: insights from organizational learning theory.
      They also shared information about organizational events, morale, and other conflicting organizational priorities with the IU and SRALab research team, which ensured the timing of the KT interventions met and did not conflict with local needs. Involving clinicians in key roles may have also increased the relevance, applicability, and ease of translation of the GAB into clinical practice.
      • Gagliardi A.R.
      • Berta W.
      • Kothari A.
      • Boyko J.
      • Urquhart R.
      Integrated knowledge translation (IKT) in health care: a scoping review.
      ,
      • Nguyen T.
      • Graham I.D.
      • Mrklas K.J.
      • et al.
      How does integrated knowledge translation (IKT) compare to other collaborative research approaches to generating and translating knowledge? Learning from experts in the field.
      ,
      • Camden C.
      • Shikako-Thomas K.
      • Nguyen T.
      • et al.
      Engaging stakeholders in rehabilitation research: a scoping review of strategies used in partnerships and evaluation of impacts.
      Importantly, this KTA project resulted in sustainable changes in practice. Recent systematic reviews on the sustainability of new practices postimplementation indicate that fewer than 50% of providers sustained practices at high levels of fidelity.
      • Wiltsey Stirman S.
      • Kimberly J.
      • Cook N.
      • Calloway A.
      • Castro F.
      • Charns M.
      The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research.
      ,
      • Ament S.M.C.
      • de Groot JJA
      • Maessen J.M.C.
      • Dirksen C.D.
      • van der Weijden T.
      • Kleijnen J.
      Sustainability of professionals’ adherence to clinical practice guidelines in medical care: a systematic review.
      Factors that may have contributed to the sustainability of the use of the GAB in practice may include the integration of the staff and consideration of the organizational context and processes into the KT plan. These are core components in the National Health Service Sustainability model, which is a diagnostic tool for identifying strengths and weaknesses in sustainability aspects of implementation plans.
      • Maher L.
      • Gustafson D.
      • Evans A.
      NHS Sustainability Model.
      ,
      • Doyle C.
      • Howe C.
      • Woodcock T.
      • et al.
      Making change last: applying the NHS Institute for Innovation and Improvement sustainability model to healthcare improvement.
      The KT plan included many critical sustainability factors, including processes such as identifying and communicating the benefits of the GAB beyond helping patients and the use of a system for monitoring progress. At a staff level, we used an integrated KT approach that involved clinicians throughout the project. The clinicians were actively involved in the codevelopment of implementation and sustainability interventions. From a context perspective, the organization was very supportive of this change. For example, the project aligned with the organization’s vision to become a learning health system. The integrated KT approach using the KTA framework allowed the development of a customized KT plan for the MFB context. Senior leaders were also highly involved, visible in the project, and undertook implementation leadership behaviors.
      • Gifford W.A.
      • Squires J.E.
      • Angus D.E.
      • et al.
      Managerial leadership for research use in nursing and allied health care professions: a systematic review.
      External facilitation also supported the organization in making these changes successful. In addition, we integrated expectations for the use of the GAB into organizational processes such as performance appraisals, job descriptions, and new hire orientation.
      Nineteen KT interventions were included in the bundle of implementation strategies used in this project. Although research demonstrates that implementation strategies are most effective when they include multiple components and target site-specific barriers, the most efficient or effective method of translating evidence into clinical practice have not been determined.
      • Jones C.A.
      • Roop S.C.
      • Pohar S.L.
      • Albrecht L.
      • Scott S.D.
      Translating knowledge in rehabilitation: systematic review.
      ,
      • Grimshaw J.M.
      • Eccles M.P.
      • Lavis J.N.
      • Hill S.J.
      • Squires J.E.
      Knowledge translation of research findings.
      • Jaeger Pedersen T.
      • Kaae Kristensen H.
      A critical discourse analysis of the attitudes of occupational therapists and physiotherapists towards the systematic use of standardised outcome measurement.
      • Stander J.
      • Grimmer K.
      • Brink Y.
      Training programmes to improve evidence uptake and utilisation by physiotherapists: a systematic scoping review.
      Use of a framework, such as the KTA framework to guide implementation may also improve the success of implementation.
      • Fischer F.
      • Lange K.
      • Klose K.
      • Greiner W.
      • Kraemer A.
      Barriers and strategies in guideline implementation-a scoping review.
      Many KT models have been studied, and little guidance is available for model selection.
      • Nilsen P.
      Making sense of implementation theories, models and frameworks.
      Organizational leaders and clinicians should evaluate different models to identify those appropriate for their context. In this study, key principles included integrated knowledge translation with engagement at all levels of the organization, use of a KT framework, and selection of a bundle of KT strategies that targeted barriers. Clinicians and leaders considering implementation projects, may consider working collaboratively with organizational stakeholders to identify barriers and implementation strategies that may be successful in their context. Providing internal funding for staff to have time to participate in the implementation project may also be beneficial.

      Study limitations

      Limitations of this study demonstrate the need for research in this area. This study used a pre-post observational study design without a control group, and we were unable to make definitive conclusions about the intervention tested. This project was implemented at a single facility and with a relatively small number of professionals. Therefore, research should be conducted to determine the effectiveness of this approach when these methods are replicated in other settings and with more practitioners. We implemented the GAB using an approach that included a bundle of KT activities that were codeveloped with clinicians and leaders and tailored to a specific facility. Research should determine adaptations required and the effectiveness of this approach when tailored to other settings. Further work should examine the contributions of using a framework, such as the KTA, as compared with specific activities in the KT bundle. In addition, research should examine the different components of this approach (ie, codevelopment of the KT plan, facilitation, implementation leadership, and tailoring of interventions) to determine the active ingredients. During the implementation of the GAB, we did not experience any staff turnover. Future research should assess sustainability after staff turnover occurs. We implemented the GAB as a standard of care but included data from patients who provided consent in the analysis. Patients who were unable to provide consent and did not have a written power of attorney were excluded, many with substantial cognitive or communication impairments. Approximately 30% of individuals with acute stroke have a cognitive impairment.
      • Sun J.H.
      • Tan L.
      • Yu J.T.
      Post-stroke cognitive impairment: epidemiology, mechanisms and management.
      Therefore, the generalization of these findings to the measurement of patients with substantial cognitive impairment could be limited. The measures selected for the GAB are well known among physical therapists. We are unsure of the effect of the clinicians’ familiarity with the measures on adherence rates. Implementation of newer or unfamiliar measures may have required different strategies, more time to implement, or may not have been implemented successfully.

      Conclusions

      We developed a comprehensive plan based on the KTA cycle to implement evidence-based gait-related measurement recommendations. An iterative approach of barrier assessment, selection of new KT interventions, and monitoring facilitated the achievement of the desired adherence levels. Successful implementation of the GAB occurred within 6 months, and the clinicians sustained these practice changes for 4 years after implementation. Active ingredients that may have contributed to the success of this project include an integrated KT approach, use of the KTA framework to guide the development of the KT plan, implementation facilitation, and leadership.

      Suppliers

      • a.
        SurveyMonkey; SurveyMonkey.
      • b.
        SPSS Statistics, version 23; IBM Corp.

      Acknowledgments

      We thank the clinicians and administrators at Mary Free Bed Rehabilitation Hospital for their contributions to data collection and knowledge translation plan development.

      Appendix

      Appendix 1KT interventions described using workgroup for intervention development and evaluation research criteria
      KT InterventionCharacteristics of Those Delivering InterventionSettingMode of DeliveryContact TimeDuration (No. of Times)Adherence/Fidelity to Delivery ProtocolsDetailed Description of Intervention Content
      Interventions Targeting Knowledge and Skill Barriers
      Consultation ∼1 mo after education sessionIU/SRALab research team and research PTsNAPhone conversation with MFB PT liaison and research coordinator1 h1NAMFB developed a list of questions after using assessments; IU/SRALab discussed answers
      Education sessionPT from IU/SRALab research teamMFB conference roomIn-person to MFB stroke team (PTs/PTAs)2 h and 50 min1Assessed % correct rating on video demonstration and standardization questionsPresentation and quiz on standardization
      In-Person discussion

      ∼8 wk after education session
      IU/SRALab PI/Co-PIMFB conference roomIn-person lunch meeting with stroke team PTs/PTAs and manager50 min1NAIU/SRALab PI/CoPI provided lunch and discussed barriers, facilitators to using the GAB; answered questions about GAB use
      Education session on test interpretationMFB PT liaisonMFB stroke unitIn-person to MFB stroke team (PTs/PTAs), e-mail50 min1Reviewed PPT with IU/SRALab CoPIDeveloped handout on interpretation of test results and goal writing and discussed with staff
      Informal discussionsIU/SRALab Co-PINAPhone conversation with MFB research coordinatorVaried from 5-60 min1 time a mo, for 15 moNAProblem solving sessions, discussions about adherence, barriers, and KT interventions
      Training of noncore staff (coverage PTs/PTAs, weekend, etc)MFB PT liaisonMFB conference room or gymIn-person90 min5 (1 per team)Used educational materials, video consistency ratings from GAB projectStandardization training for 5 additional MFB teams (Brain Injury, Medically Complex Team, Sub-Acute Rehab Team, Core Weekend PTs/PTAs, coverage services PT/PTAs), using same training methods and tested for consistency using videos PT and PTA students also trained.
      Restandardization 1 year after implementationMFB

      PT liaison
      MFB conference roomIn-person lunch meeting with stroke team PTs/PTAs and manager1 h1Assessed % correct rating on video demonstration and standardization questionsPresentation of adherence data for group and individual PT staff; discussion of barriers and facilitators for adherence; review of standardization items with which clinicians had the most difficulty
      Development of “cheat sheets”MFB PT liaisonMFB stroke unitDistributed via email, discussed in meetings1-h discussion2 (initial, updated∼1 y later)NAInformation to support test interpretation in subacute stroke (provided online)
      Interventions Targeting Social Influences
      Obtain leadership supportIU/SRALab PI/CoPI and MFB research teamMFBIn-person and phone meetings∼8 h2NADiscussion of the project, expectations, and suggestions for adherence and to demonstrate leadership support; discussion with stroke team manager about processes and equipment to support program, and with Medical Director to require GAB reporting weekly
      Articulate leadership supportMFB stroke unit manager, physicians, and executivesMFBIn-person during team meetings and conferences, and e-mails1 h3NARe-enforced project importance, adherence expectations, and reporting of results in team conference
      Required team conference reportingMFB research and leadership teamMFB, during team conferenceE-mailed expectations, in-person during team conference when reporting did not occurReporting is weekly, monitored in-person by leadership and physiciansIntermit-tent for 12 moReporting requirements monitored by physicians and managersE-mails and in-person encouragement for MFB clinicians to report patient test results and changes demonstrated in outcome measurement
      Testing “Tuesday”MFB research team and leadershipMFB stroke unitIn-person, by e-mailWeekly during therapy sessionOngoing, integrated into processesNATuesdays were designated the regular testing days; clinicians were notified by e-mails and team meetings; initially reminded by research assistant; clinicians and patients continue to remind each other/support use
      Audit and feedbackMFB research team e-mailed, cc’d managerNAE-mail to the entire group of PTs/PTAs, in-person meetings3 in-person 1-h meetings, other times by e-mailFeedback provided every 6 wk for 15 moNADetailed in appendix
      RewardsMFB Leadership Team (Director of Research)MFB team meetingIn-person, provided food and awards1 h2NADirector of Research provided 2 lunches and rewarded staff for high adherence; Most improved award was also provided
      Interventions Targeting Environmental Context and Resources
      Equipment purchaseMFB research teamMFB stroke unitPurchased and stored in rehabilitation gymNA1NAYardsticks, measuring wheels, file folders; Plant Services department painted baseboards in halls for 10MWT
      Research assistant support for measurementMFB research assistantMFB Stroke Unit: gymIn-personAvailable 8 am-4 pm on “Testing Tuesday” to support test administration6 moNAResearch assistant was available in the gym to assist with testing (ie, set up equipment, obtain testing forms, and assist with testing as needed)
      Data collection formsData collection forms developed by IU/SRALab research team, revised by MFB team; clinicians provided feedback, which was integrated into each versionDeveloped at each site, used during testing at MFBWord documentsForms reviewed during initial education session; ongoing verbal and e-mail communication with PTs to optimize form1, then stored in gym; revisions occurred 10 times and were communicated by e-mail and verballyAssessment forms were reviewed by MFB research teamData collection forms can be reviewed online; data collection forms were stored alphabetically in file folder in main gym; updated weekly; uploaded into electronic health record after patient discharge
      Scanning of data collection formsMFB research assistantMFBElectronic: data collection forms scanned into medical recordForms scanned weekly, ∼1 hOngoingNARA scans paper data collection forms into the medical chart after discharge
      Abbreviations: CoPI, co-principal investigator; NA, not applicable; PI, principal investigator; PT, physical therapist; PTA, physical therapist assistant; RA, research assistant.
      Appendix 2Audit and feedback intervention design
      Design ElementDescription
      Who?
      1. Was the feedback given to an individual, a group, or both?Provided to the group, reporting individual data
      2. Was it given to the person in whom the practice change was desired (eg, healthcare provider vs hospital administrator)?Provided to the clinicians directly, cc’d manager
      What?
      3. Was there feedback about the processes of care?Yes, feedback about the rate of compliance with administration recommendations (actual administration/expected administration)
      4. Was there feedback about patient outcomes?No
      5. Was there feedback about something other than processes of care or patient outcomes?No
      6. Was the feedback about individual provider performance?Yes, individual level data reported to the whole group
      7. Was the feedback about the performance of the provider group?Yes, performance of the group was also provided
      8. Was the feedback about individual patient cases?No
      9. Was the feedback about an aggregate of patient cases?Yes, all patients discharged in the last month
      10. Did the feedback identify a specific behavior(s) to be changed?Yes, assessment administration rate
      11. What was the comparison provided in the feedback?Yes, historical administration rates and target rate
      12. Were graphical elements included in the feedback?Graphical feedback was provided at team meetings, 3 times in year 1
      When?
      13. What was the lag between the time of the audit and the delivery of the feedback?First feedback delivered 4 mo after staff training; feedback delivered approximately every 2 mo thereafter
      Why?
      14. What is the rationale for using audit and feedback?To monitor and improve compliance with assessment recommendations using clear communication with the team and manager
      Who?
      15. Was the feedback given face to face?Feedback was given face to face 3 times: 2 times when compliance was low and once at the 1-y restandardization meeting; the remaining times were provided by e-mail or verbal communication
      16. Were providers explicitly asked to consider the implications the audit and feedback had for their practice?No
      How much?
      17. What was the total number of times the feedback was given?8 times
      NOTE. The intervention is described according to the modifiable design elements as reported by Colquhoun et al.
      • Colquhoun H.
      • Michie S.
      • Sales A.
      • et al.
      Reporting and design elements of audit and feedback interventions: a secondary review.

      References

      1. Guide to physical therapist practice: measurement and outcomes.
        (Available at:)
        http://guidetoptpractice.apta.org/content/1/SEC3.body
        Date: 2014
        Date accessed: December 21, 2017
      2. Smith M. Saunders R. Stuckhardt L. McGinnis J.M. Best care at a lower cost: the path to continuously learning health care in America. The National Acadamies Press, Washington DC2013
        • Potter K.
        • Cohen E.T.
        • Allen D.D.
        • et al.
        Outcome measures for individuals with multiple sclerosis: recommendations from the American Physical Therapy Association Neurology Section task force.
        Phys Ther. 2014; 94: 593-608
        • Sullivan J.E.
        • Crowner B.E.
        • Kluding P.M.
        • et al.
        Outcome measures for individuals with stroke: process and recommendations from the American Physical Therapy Association neurology section task force.
        Phys Ther. 2013; 93: 1383-1396
        • Kahn J.H.
        • Tappan R.
        • Newman C.P.
        • et al.
        Outcome measure recommendations from the Spinal Cord Injury EDGE Task Force.
        Phys Ther. 2016; 96: 1832-1842
        • McCulloch K.L.
        • de Joya A.L.
        • Hays K.
        • et al.
        Outcome measures for persons with moderate to severe traumatic brain injury: recommendations from the American Physical Therapy Association Academy of Neurologic Physical Therapy TBI EDGE Task Force.
        J Neurol Phys Ther. 2016; 40: 269-280
      3. ANPT outcome measures recommendations.
        (Available at:)
        • Moore J.L.
        • Potter K.
        • Blankshain K.
        • Kaplan S.L.
        • O'Dwyer L.C.
        • Sullivan J.E.
        A core set of outcome measures for adults with neurologic conditions undergoing rehabilitation: a clinical practice guideline.
        J Neurol Phys Ther. 2018; 42: 174-220
        • Winstein C.J.
        • Stein J.
        • Arena R.
        • et al.
        Guidelines for adult stroke rehabilitation and recovery: a guideline for healthcare professionals from the American Heart Association/American Stroke Association.
        Stroke. 2016; 47 (e98-169)
        • Jette D.U.
        • Halbert J.
        • Iverson C.
        • Miceli E.
        • Shah P.
        Use of standardized outcome measures in physical therapist practice: perceptions and applications.
        Phys Ther. 2009; 89: 125-135
        • Swinkels R.A.
        • van Peppen R.P.
        • Wittink H.
        • Custers J.W.
        • Beurskens A.J.
        Current use and barriers and facilitators for implementation of standardised measures in physical therapy in the Netherlands.
        BMC Musculoskelet Disord. 2011; 12: 106
        • Colquhoun H.L.
        • Lamontagne M.E.
        • Duncan E.A.
        • Fiander M.
        • Champagne C.
        • Grimshaw J.M.
        A systematic review of interventions to increase the use of standardized outcome measures by rehabilitation professionals.
        Clin Rehabil. 2017; 31: 299-309
        • Duncan E.A.
        • Murray J.
        The barriers and facilitators to routine outcome measurement by allied health professionals in practice: a systematic review.
        BMC Health Serv Res. 2012; 12: 96
      4. Canadian Institutes of Health Research.
        (Available at:)
        http://www.cihr-irsc.gc.ca/e/29418.html
        Date accessed: January 7, 2016
        • Russell D.J.
        • Rivard L.M.
        • Walter S.D.
        • et al.
        Using knowledge brokers to facilitate the uptake of pediatric measurement tools into clinical practice: a before-after intervention study.
        Implement Sci. 2010; 5: 92
        • Schreiber J.
        • Marchetti G.F.
        • Racicot B.
        • Kaminski E.
        The use of a knowledge translation program to increase use of standardized outcome measures in an outpatient pediatric physical therapy clinic: administrative case report.
        Phys Ther. 2015; 95: 613-629
        • Fischer F.
        • Lange K.
        • Klose K.
        • Greiner W.
        • Kraemer A.
        Barriers and strategies in guideline implementation-a scoping review.
        Healthcare (Basel). 2016; 4: 36
        • Graham I.D.
        • Logan J.
        • Harrison M.B.
        • et al.
        Lost in knowledge translation: time for a map?.
        J Contin Educ Health Prof. 2006; 26: 13-24
        • Straus S.
        • Tetroe J.
        • Graham I.
        Knowledge Translation in health care: moving from evidence to practice.
        2nd ed. BMJ Books, West Sussex, UK2013
        • Kothari A.
        • McCutcheon C.
        • Graham I.D.
        Defining integrated knowledge translation and moving forward: a response to recent commentaries.
        Int J Health Policy Manag. 2017; 6: 299-300
        • Shea C.M.
        • Jacobs S.R.
        • Esserman D.A.
        • Bruce K.
        • Weiner B.J.
        Organizational readiness for implementing change: a psychometric assessment of a new measure.
        Implement Sci. 2014; 9: 7
        • Atkins L.
        • Francis J.
        • Islam R.
        • et al.
        A guide to using the Theoretical Domains Framework of behaviour change to investigate implementation problems.
        Implement Sci. 2017; 12: 77
        • Michie S.
        • Johnston M.
        • Francis J.
        • Hardeman W.
        • Eccles M.
        From theory to intervention: mapping theoretically derived behavioural determinants to behaviour change techniques.
        Appl Psychol. 2008; 57: 660-680
        • Butzer J.F.
        • Virva R.
        • Lenca L.
        Commentary on the challenges and benefits of implementing standardized outcome measures.
        Arch Phys Med Rehabil. 2020 Nov 25; ([Epub ahead of print])
        • Blum L.
        • Korner-Bitensky N.
        Usefulness of the Berg Balance Scale in stroke rehabilitation: a systematic review.
        Phys Ther. 2008; 88: 559-566
        • Straube D.
        • Moore J.
        • Leech K.
        • Hornby T.G.
        Item analysis of the berg balance scale in individuals with subacute and chronic stroke.
        Top Stroke Rehabil. 2013; 20: 241-249
        • Fulk G.D.
        • Echternach J.L.
        Test-retest reliability and minimal detectable change of gait speed in individuals undergoing rehabilitation after stroke.
        J Neurol Phys Ther. 2008; 32: 8-13
        • Fulk G.D.
        • Echternach J.L.
        • Nof L.
        • O'Sullivan S.
        Clinometric properties of the six-minute walk test in individuals undergoing rehabilitation poststroke.
        Physiother Theory Pract. 2008; 24: 195-204
        • Hornby T.G.
        • Holleran C.L.
        • Leddy A.L.
        • et al.
        Feasibility of focused stepping practice during inpatient rehabilitation poststroke and potential contributions to mobility outcomes.
        Neurorehabil Neural Repair. 2015; 29: 923-932
        • Colquhoun H.L.
        • Carroll K.
        • Eva K.W.
        • et al.
        Advancing the literature on designing audit and feedback interventions: identifying theory-informed hypotheses.
        Implement Sci. 2017; 12: 117
        • Colquhoun H.
        • Michie S.
        • Sales A.
        • et al.
        Reporting and design elements of audit and feedback interventions: a secondary review.
        BMJ Qual Saf. 2017; 26: 54-60
        • Albrecht L.
        • Archibald M.
        • Arseneau D.
        • Scott S.D.
        Development of a checklist to assess the quality of reporting of knowledge translation interventions using the Workgroup for Intervention Development and Evaluation Research (WIDER) recommendations.
        Implement Sci. 2013; 8: 52
        • Colquhoun H.L.
        • Squires J.E.
        • Kolehmainen N.
        • Fraser C.
        • Grimshaw J.M.
        Methods for designing interventions to change healthcare professionals' behaviour: a systematic review.
        Implement Sci. 2017; 12: 30
        • Jones C.A.
        • Roop S.C.
        • Pohar S.L.
        • Albrecht L.
        • Scott S.D.
        Translating knowledge in rehabilitation: systematic review.
        Phys Ther. 2015; 95: 663-677
        • Ivers N.
        • Jamtvedt G.
        • Flottorp S.
        • et al.
        Audit and feedback: effects on professional practice and healthcare outcomes.
        Cochrane Database Syst Rev. 2012; 6: CD000259
        • Ivers N.M.
        • Grimshaw J.M.
        • Jamtvedt G.
        • et al.
        Growing literature, stagnant science? Systematic review, meta-regression and cumulative analysis of audit and feedback interventions in health care.
        J Gen Intern Med. 2014; 29: 1534-1541
        • Gagliardi A.R.
        • Berta W.
        • Kothari A.
        • Boyko J.
        • Urquhart R.
        Integrated knowledge translation (IKT) in health care: a scoping review.
        Implement Sci. 2016; 11: 38
        • Nguyen T.
        • Graham I.D.
        • Mrklas K.J.
        • et al.
        How does integrated knowledge translation (IKT) compare to other collaborative research approaches to generating and translating knowledge? Learning from experts in the field.
        Health Res Policy Syst. 2020; 18: 35
        • Gifford W.A.
        • Davies B.
        • Edwards N.
        • Graham I.D.
        Leadership strategies to influence the use of clinical practice guidelines.
        Nurs Leadersh (Tor Ont). 2006; 19: 72-88
        • Harlos K.
        • Tetroe J.
        • Graham I.D.
        • Bird M.
        • Robinson N.
        Mining the management literature for insights into implementing evidence-based change in healthcare.
        Healthc Policy. 2012; 8: 33-48
        • Salbach N.M.
        • Jaglal S.B.
        • Korner-Bitensky N.
        • Rappolt S.
        • Davis D.
        Practitioner and organizational barriers to evidence-based practice of physical therapists for people with stroke.
        Phys Ther. 2007; 87: 1284-1303
        • Berta W.
        • Cranley L.
        • Dearing J.W.
        • Dogherty E.J.
        • Squires J.E.
        • Estabrooks C.A.
        Why (we think) facilitation works: insights from organizational learning theory.
        Implement Sci. 2015; 10: 141
        • Harvey G.
        • Kitson A.
        Implementing evidence-based practice in healthcare. a facilitation guide.
        Routledge, London2015
        • Camden C.
        • Shikako-Thomas K.
        • Nguyen T.
        • et al.
        Engaging stakeholders in rehabilitation research: a scoping review of strategies used in partnerships and evaluation of impacts.
        Disabil Rehabil. 2015; 37: 1390-1400
        • Wiltsey Stirman S.
        • Kimberly J.
        • Cook N.
        • Calloway A.
        • Castro F.
        • Charns M.
        The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research.
        Implement Sci. 2012; 7: 17
        • Ament S.M.C.
        • de Groot JJA
        • Maessen J.M.C.
        • Dirksen C.D.
        • van der Weijden T.
        • Kleijnen J.
        Sustainability of professionals’ adherence to clinical practice guidelines in medical care: a systematic review.
        BMJ Open. 2015; 5e008073
        • Maher L.
        • Gustafson D.
        • Evans A.
        NHS Sustainability Model.
        NHS Institute for Innovation and Improvement, 2010 (Available at: www.institute.nhs.uk/sustainability. Accessed December 19, 2020)
        • Doyle C.
        • Howe C.
        • Woodcock T.
        • et al.
        Making change last: applying the NHS Institute for Innovation and Improvement sustainability model to healthcare improvement.
        Implement Sci. 2013; 8: 127
        • Gifford W.A.
        • Squires J.E.
        • Angus D.E.
        • et al.
        Managerial leadership for research use in nursing and allied health care professions: a systematic review.
        Implement Sci. 2018; 13: 127
        • Grimshaw J.M.
        • Eccles M.P.
        • Lavis J.N.
        • Hill S.J.
        • Squires J.E.
        Knowledge translation of research findings.
        Implement Sci. 2012; 7: 50
        • Jaeger Pedersen T.
        • Kaae Kristensen H.
        A critical discourse analysis of the attitudes of occupational therapists and physiotherapists towards the systematic use of standardised outcome measurement.
        Disabil Rehabil. 2016; 38: 1592-1602
        • Stander J.
        • Grimmer K.
        • Brink Y.
        Training programmes to improve evidence uptake and utilisation by physiotherapists: a systematic scoping review.
        BMC Med Educ. 2018; 18: 14
        • Nilsen P.
        Making sense of implementation theories, models and frameworks.
        Implement Sci. 2015; 10: 53
        • Sun J.H.
        • Tan L.
        • Yu J.T.
        Post-stroke cognitive impairment: epidemiology, mechanisms and management.
        Ann Transl Med. 2014; 2: 80