The Use of the Analytic Hierarchy Process to Aid Decision Making in Acquired Equinovarus Deformity

Article Outline

• Abstract
• Methods
  • Model Development
  • Judgment Stage
  • Synthesis of Results
  • Sensitivity Analysis
• Results
  • Criteria Weights
  • Performance Weights
  • Priority of Treatments
  • Sensitivity Analysis
• Discussion
• Conclusions
• Appendix 1. Questionnaire Stroke Interest Group
• Appendix 2. Performance and Importance Judgment
  • Performance
  • Importance
• References
• Copyright

Abstract 

van Til JA, Renzenbrink GJ, Dolan JG, IJzerman MJ. The use of the analytic hierarchy process to aid decision making in acquired equinovarus deformity.

Objective

To increase the transparency of decision making about treatment in patients with equinovarus deformity poststroke.

Design

The analytic hierarchy process (AHP) was used as a structured methodology to study the subjective rationale behind choice of treatment.

Setting

An 8-hour meeting at a centrally located rehabilitation center in The Netherlands, during which a patient video was shown to all participants (using a personal computer and a large screen) and the patient details were provided on paper.

Participants

A panel of 10 health professionals from different backgrounds.

Interventions

Not applicable.

Main Outcome Measures

The performance of the applicable treatments on outcome, impact, comfort, cosmetics, daily effort, and risks and side effects of treatment, as well as the relative importance of criteria in the choice of treatment.

Results

According to the model, soft-tissue surgery (.413) ranked first as the preferred treatment, followed by orthopedic footwear (.181), ankle-foot orthosis (.147), surface electrostimulation (.137), and finally implanted electrostimulation (.123). Outcome was the most influential consideration affecting treatment choice (.509), followed by risk and side effects (.194), comfort (.104), daily effort (.098), cosmetics (.065), and impact of treatment (.030).

Conclusions

Soft-tissue surgery was judged best on outcome, daily effort, comfortable shoe wear, and cosmetically acceptable result and was thereby preferred as a treatment alternative by the panel in this study. In contrast, orthosis and orthopedic footwear are usually preferred in daily practice. The AHP method was found to be suitable methodology for eliciting subjective opinions and quantitatively comparing treatments in the absence of scientific evidence.

Key Words: Decision making, Equinovarus, Rehabilitation, Stroke

A STROKE, OR A cerebrovascular accident, is a common disorder that can result in paralysis of 1 side of the body. About 20% of stroke patients are confronted with a deviant position of the ankle and foot that hinders standing and walking. The impairment is known as an equinovarus deformity. In 2003, the International Society for Prosthetics and Orthotics developed guidelines for the orthotic management of stroke patients.¹, ² The guidelines provide insight in the available treatments for equinovarus deformity, but no support is given on the decision for treatment if multiple treatments are available. Although the specifics of the deformity influence the availability of treatments, in general, surgical,³, ⁴ technologic,⁵ pharmaceutical,⁶ and orthotic⁷ treatments are available to correct the deformity.⁸ Treatment is primarily aimed at correcting the deviant foot position. Secondary beneficial effects include reduction or stimulation of muscle force, improvement of walking speed and distance, and reduction of energy consumption.

Evidence-based choice of treatment is hindered because the available clinical evidence base consists of only a few articles with small sample size and poor methodologic quality.², ³, ⁵, ⁹

In clinical practice, however, the decision for treatment of equinovarus deformity has to be made on a daily basis in the absence of convincing evidence. Treatment decisions in acquired neurologic equinovarus deformity can thereby be described as a preference-sensitive, or equipoise, decision, where choice is based on personal preferences and convictions rather than evidence.¹⁰ According to Frith¹¹ the perceived attractiveness of treatment involves many subjective elements in addition to potential effectiveness. The subjective preferences of clinicians in daily practice therefore might hold valuable information on the strengths and weaknesses of treatment alternatives.

It is difficult to elicit and measure subjective preferences in health care decision making. The analytic hierarchy process (AHP) is an approach to multicriteria decision making problems of choice and prioritization. The AHP has been previously used to compare the performance of intensive care units,¹² allocate livers among transplant patients,¹³ select tests for abdominal pain,¹⁴ and support the choice of reconstructive interventions in spinal cord injury.¹⁵ The AHP evaluates decision problems by determining the relative importance of criteria and the performance of available solutions through a series of trade-offs.¹⁶ The comparison can include medical, technical, economic, and social aspects. Subjective judgments on aspects of a problem for which no scale of measurement exists can be easily accommodated.¹⁷ The aim of this study was to test the AHP as a methodology for studying decision making regarding choice of treatment in rehabilitation medicine in equinovarus deformity.

Back to Article Outline

Methods 

An AHP analysis can be divided into 4 stages, namely, (1) model development, (2) judgment stage, (3) synthesis of results, and (4) sensitivity analysis. For a more detailed discussion on the technique and underlying methodology, we refer the reader to the literature.¹⁶, ¹⁸, ¹⁹

Model Development 

The AHP decision model in this study consists of 4 levels (fig 1). In the first level, the goal of the analysis is presented. The second level consists of the main decision criteria. These criteria are split into subcriteria, which are included in the third level. Finally, the available alternatives are presented in the fourth level of the model.

• 
  • View Large Image
  • Download to PowerPoint

• Fig 1. 

  The AHP decision model. Impact of treatment comprises the impact to the patient in the active treatment phase, and daily effort comprises the daily actions needed to maintain the effect of treatment afterward (mainly donning and doffing necessary equipment). *Criteria supported by literature.²⁶

The goal of the analysis was set at selecting the preferred treatment for equinovarus deformity after stroke. This goal supports the specific aim of the study, which was to obtain information on the preferred treatment of a selection of health care professionals for equinovarus deformity in stroke. To determine level 2 and 3 criteria, a Delphi-style paper-and-pencil interview was conducted among a sample of Dutch physiatrists (n=28; response rate, 68%) united in a stroke interest group. The questionnaire was developed to identify a preliminary set of treatment requirements, criteria, and treatment alternatives for acquired equinovarus deformity in a postacute phase (>6wk) of stroke. An example of the first questionnaire as it was sent to the group is presented in appendix 1. Simultaneously, a literature search was conducted using the search terms stroke, equinovarus deformity, ankle-foot impairment, and treatment.³, ⁴, ⁵, ⁶, ⁷ A summary of responses to the initial questionnaire was sent back to participants, along with the treatments reported in literature and 2 additional rounds of feedback from the group were processed. After 3 rounds of questionnaires, 5 feasible treatment categories for the postacute management of equinovarus foot were identified from the treatments selected by the panel based on the frequency with which they were mentioned. These treatments were soft-tissue surgery, functional electric stimulation (FES), ankle-foot orthosis (AFO), pharmaceutical treatment, and orthopedic shoes. A decision tree was also developed based on the group responses.

Judgment Stage 

A panel of 10 professionals in rehabilitation medicine was asked to participate in the judgment phase. The panel consisted of 4 physiatrists, 1 orthopedic surgeon, 1 physical therapist, 2 senior researchers in the field of stroke, and 2 certified orthotists. To start the discussion, a patient case description (fig 2) and video were presented to the panel. The patient had moderate equinovarus deformity and was chosen so that it could be treated with the maximal number of treatments identified during the Delphi process. Based on their individual expertise, the panel members supported the use of the following treatments for equinovarus deformity: split tibial tendon transfer combined with a transfer of the hallucis longus muscle (soft-tissue surgery), an off-the-shelf AFO, off-the-shelf semi-orthopedic shoes, an implanted peroneal stimulator, and a surface peroneal stimulator. Pharmaceutical treatment was omitted as a treatment alternative because of the absence of marked spasticity.

• 
  • View Large Image
  • Download to PowerPoint

• Fig 2. 

  Patient case description. Observations were based on extensive manual testing in lying and sitting positions and visual observation of the patient during walking by an experienced physiotherapist. Abbreviations: FAC, Functional Ambulation Category; MRC, Medical Research Council scale.

The decision model was presented to the panel on a projection screen, which stood in the front of the room.

The judgment stage consists of 2 phases. First, the treatment performance was judged on the level 3 subcriteria. These judgments were made by comparing the treatments in a pair-wise fashion using a reciprocal numeric scale ranging from 1 to 9 (an example can be found in appendix 2). In the AHP, the numbers are associated with verbal statements ranging from equally preferred (1) to extremely more preferred (9).

In the next phase of the judgment stage, the panel members were first asked to judge the importance of the level 3 subcriteria with regard to the accomplishment of the level 2 criteria and then to judge the contribution (importance) of level 2 criteria in the achievement of the goal. These judgments were also made in a pairwise fashion using the reciprocal numeric scale described earlier using verbal statements ranging from equally important (1) to extremely more important (9).

Initially, individual judgments were made by the panel members. When all judgments were completed, disagreement among panel members was made explicit by presenting the numeric judgments on a screen visible to each panel member and by verbal explanation of the discussion leader. In case of disagreement, panel members with the most diverging scores were asked to clarify their judgments. In many cases, this resulted in a discussion of pros and cons of the treatments or the relevance of criteria. Panel members were allowed to reconsider and to alter their judgments if they believed it was necessary based on the discussion.

To reduce the time requirements of the session, the number of performance and importance judgments was reduced to the minimal number required for calculating weights on an individual and group level. A total of 52 performance and 13 importance judgments were made by the 10 members of the panel. There was a group discussion after each judgment. The duration of the session was about 7 hours (including breaks).

Synthesis of Results 

In the AHP, the numeric judgments are put in a comparison matrix (A). The reciprocal scores are used to complete the matrix. The principle eigenvector method is used to calculate the individual importance and performance weights from the matrix. For a detailed explanation of the hows and whys of the eigenvector method the reader is referred to the literature,¹⁶, ²⁰ but for the scope of the current article it is enough to know that a close approximation of the priority vector of matrix A can be calculated by dividing each element in the matrix by the sum of its column and then dividing the sum of each row by the sum of the matrix. Group values are calculated using the geometric mean of individual judgments.

A priority score (V) for each treatment (a) is calculated based on an additive value function:

Criteria weights are multiplied with performance values and summed, resulting in an overall priority (performance rating) for each alternative.

Measurement of the consistency of judgments is a routine part of the AHP and is measured by calculating a consistency index, which ranges from 0 to 1. Analyses with consistency indices less than or equal to 0.1 are generally considered acceptable. In certain cases, a cutoff value of 0.2 is used. In this study, we calculated consistency for the entire group using the group mean comparisons as described earlier.

Sensitivity Analysis 

To determine the impact of importance weights on overall priority for treatment, the importance of each level 2 criterion was varied from 0 to 1, in increments of 0.1, and the sum of importance weights was kept at 1.0. The priority for treatment was calculated for all combinations.

Back to Article Outline

Results 

Criteria Weights 

The relative priorities of the level 2 and 3 criteria are presented in table 1. In the opinion of the expert panel, on the second level the outcome of treatment is the most important criterion in making treatment decisions. With an importance weight of .509 it influences half of the decision. Active ankle stability (.44) and foot position in swing phase of walking (.4) were judged to be the most important level 3 parameters when outcome is considered, whereas passive ankle stability and improvement of the foot position in stance were judged less important.

Table 1. Importance Weights for All Criteria

Abbreviation: SD, standard deviation.

Performance Weights 

The performance values for all treatments are presented in figure 3.

• 
  • View Large Image
  • Download to PowerPoint

• Fig 3. 

  The average performance values with standard deviation of the treatments on all level 3 criteria. IFES, implanted peroneal stimulator; SFES, surface peroneal stimulator; SOS, semi-orthopedic shoes; TS, soft-tissue surgery.

Priority of Treatments 

Soft-tissue surgery was the first ranked treatment in all participants, with an average priority of .413 and a range between .313 and .614. Semi-orthopedic shoes (.181; range, .081–.230) ranked second, followed by the AFO (.147; range, .098–.170), surface peroneal stimulator (.137; range, .079–.230), and finally implanted peroneal stimulator (.123; range, .099–.169). Individual priority scores along with participants’ professions are presented in table 2.

Table 2. Indvidual Priority for Treatment

Abbreviations: IFES, implanted peroneal stimulator; SFES, surface peroneal stimulator; SOS, semi-orthopedic shoes; TS, soft-tissue surgery.

Sensitivity Analysis 

Sensitivity analysis was performed to study the influence of criteria weights on treatment ranking. The priority for treatment is pictured in figure 4. The first ranked treatment was soft-tissue surgery in 77%, orthopedic shoes in 3%, and AFO in 20% of cases. No combination of importance weights results in either surface or implanted electric stimulation having the highest priority.

• 
  • View Large Image
  • Download to PowerPoint

• Fig 4. 

  Sensitivity analysis. A 6-dimensional space is presented by showing each combination of weights for 2 criteria in 16 squares. If the preference for treatment is dependent on the weight of other criteria, multiple treatment priorities are piled for a 2-dimensional weight combination.

Back to Article Outline

Discussion 

Using the AHP methodology, soft-tissue surgery was the preferred treatment in this study. The panel judged soft-tissue surgery performance best on outcome, cosmetics, comfort, and daily effort. Orthopedic shoes and AFOs perform modestly on most criteria. Strong features include the low impact of treatment and the low risk that is associated with treatment in both orthopedic shoes and AFOs. The performance of FES was judged low. As was shown in the sensitivity analysis, FES was dominated by the other treatment alternatives for equinovarus deformity, because no combination of criteria weights resulted in a priority for FES. According to the expert panel, problems with surface FES include the complexity of electrode and heel switch placement in a patient group that can be both cognitively and/or physically impaired. Although an effort to reduce daily effort was made by the development of an implantable device, the surgical procedure negatively influences treatment impact and risks and side effects. In terms of outcome, the timing of the electric stimulus, which is the signal for foot elevation during the swing phase of walking, is thought to be unreliable. Naturally, as a result of ongoing development in rehabilitation medicine the conclusions of this experiment regarding preference for treatment are temporary, and updating the model with new and if possible objective information as it becomes available would be essential.²¹

The preference for soft-tissue surgery expressed in this study is not in accordance with clinical practice. In the Netherlands, orthopedic shoes and AFOs are used most often in the treatment of equinovarus deformity. Panel discussion suggested that this might be explained by the low impact of these treatments in patients who are faced with many cognitive and physical disabilities and a great familiarity with and easy access to orthotic treatment in a rehabilitation setting. According to the panel the preference for soft-tissue surgery in this study is partly explained by the time since injury (>1y) and the good functional status of the patient in the case description. As a direct consequence, treatment impact is judged to be least and outcome is considered most important. In clinical practice the natural recovery of equinovarus deformity and walking ability is often maximal within weeks, and a decision regarding treatment has to be made at a time in which the treatment of equinovarus deformity as such might not have the highest priority. However, as time progresses, different trade-offs between the pros and cons of treatment are made. Another explanation for the limited use of soft-tissue surgery might be the limited experience of neurologists and physiatrists, who traditionally treat stroke patients, with the surgical possibilities in an adult population with upper motoneuron disorders. Consequently stroke patients are only rarely referred to orthopedic surgeons, although the results of this study indicate there is a high potential for the use of soft-tissue surgery in the later stages of recovery. This advocates a more active approach in bringing this often-neglected or unknown surgical approach in equinovarus deformity in stroke to the attention of treating physicians.

In our opinion, the AHP methodology that was used in this study provided the condition for a structured discussion regarding the conditions for treatment decisions for equinovarus deformity in stroke.¹⁷, ²² The conscious deliberation of relevant requirements for treatments gave insight into the thought processes of panel members. The judgment of performance on a standard set of well-defined criteria enabled a comparison of treatments that was not possible based on the available literature. Individual knowledge and experience was made explicit and was shared among panel members.

The reaction of the panel to the use of the AHP as an alternative tool for treatment comparison was diverse. During the analysis, some of the panel members raised concerns about the validity of the judgments because their experience and knowledge on some of the treatments was limited. In our opinion, the group discussion phase should have resolved these problems, because panel members were allowed to alter their judgments if they believed unknown information was presented. Moreover, the inclusion of and discussion between panel members from different backgrounds is essential in the comparison of conventional and innovative or specialized treatment options, because any individual’s knowledge and experience are always restricted.

The time requirements to complete the model were considered a disadvantage, and the AHP was thought to be bothersome for use in day-to-day decision making. This reservation might be a result of the large size of the current model. Simpler models might be possible, although care must be taken to maintain important details. The AHP analysis gave valuable insights in the motivation of panel members and is thereby very well suited for both examination of treatment choice and guideline development in rehabilitation medicine when high-quality scientific evidence is unavailable.

Retrospectively, panel members expressed concern regarding the generalizability of the results of the study to other patients because the judgment phase may have been influenced by the patient description used. We shall discuss these concerns on 2 levels—namely, the effect on performance values of the treatment alternatives and the effect on importance weights.

First, the influence of the case description on the performance judgments of the expert panel was negligible. As became obvious from the videotape used to document the experiment, the patient description was rarely referred to by the panel members in the discussion, and judgments on performance were made on a more general level. Moreover, because the influence of minor variations in impairment on treatment performance on treatment outcome is only scarcely documented in literature, it is questionable whether it would become obvious from subjective judgments made by a panel of health professionals. For a number of other criteria, the effect of the case description would be comparable for all treatments (eg, more severe deformity would lead to higher treatment impact), which in turn does not influence the comparison. In our opinion, therefore, the judgments on treatment performance on the lowest level are generalizable to other patients as well. Second, as was mentioned before, the specifics of the ankle-foot deformity and the time since injury probably did influence the panel’s judgments about the relative importance of the criteria on a higher level. In this patient, the relevance of treatment impact was judged to be negligible and outcome was deemed most important as a result of the specific status and demands of the patient. Because the impairment was especially bothersome in the swing phase of walking, the outcome of treatments on criteria related to the swing phase of walking were given a high importance weight. Criteria and subcriteria importance should therefore be determined for each patient to determine the preference for treatment in other cases. To support these statements, it would be worthwhile to repeat the analysis with another patient and/or another panel.

The high importance that is given to health-related criteria such as outcome and risks of treatment may be a reflection of the preferences of the health professionals represented in the panel. As was shown by Hummel et al,¹⁵ patients value different aspects of treatment compared with health professionals. Treatment preference could thereby differ for patients. As patient involvement is seen as an important factor in ensuring quality improvement in health care,²³, ²⁴ health professionals are increasingly encouraged to involve their patients in treatment decision making. The AHP model used in this study could be a way to include the personal aims, wishes, and demands of each patient into health care decision making. It has been shown previously¹⁵, ²⁵ that patients are capable of using the AHP to express their preferences.

Back to Article Outline

Conclusions 

When clinical practice is compared with the preference for treatment expressed using the AHP methodology, soft-tissue surgery is an undervalued treatment alternative in the treatment of moderate equinovarus deformity in the swing phase of walking, if a patient is willing and able to undergo a longer and more tedious treatment. However, the final choice of treatment remains dependent on individual trade-offs of patients and health care professionals, and this study showed that different trade-offs regarding criteria importance could lead to a different preference for treatment.

Although some concerns were raised by the panel, these seem manageable, and the use of the AHP as a methodology to compare treatment alternatives in rehabilitation medicine is promising. Because large, randomized, controlled trials are not yet available and will remain difficult to execute because of the heterogeneity of the patient population and lack of funding in rehabilitation medicine, an alternative for decision aiding is highly applicable.

Back to Article Outline

Appendix 1. Questionnaire Stroke Interest Group 

An open-ended questionnaire with the following questions was sent to the participants.

All responses were categorized and returned to participants. Participants were asked to check the boxes with the treatments, characteristics, or criteria that they found relevant in the decision for treatment. Based on frequency a selection of relevant criteria was made for use in the decision tree. Also, a selection of alternative treatments was made.

Back to Article Outline

Appendix 2. Performance and Importance Judgment 

Performance 

Which of the next treatments is preferred when the aim is to minimize risks?

Soft-tissue surgeryorAnkle-foot orthosis

Importance 

Which of the next criteria is more important in determining the best treatments for ankle-foot impairments?

outcomeorrisks

Back to Article Outline

References 

1. Geertzen JH, Hijmans JM, van der Linde H. Prosthetic prescription in The Netherlands: an interview with clinical experts. Prosthet Orthot Int. 2005;29:113–114
   • View In Article
   • MEDLINE
   • CrossRef
2. Report of a consensus conference on the orthotic management of stroke patients. Copenhagen: International Society of Prosthetics and Orthotics; 2004;
   • View In Article
3. Piccioni L, Keenan M. Surgical correction of varus and equinovarus deformity in the spastic patient. Oper Tech Orthop. 1992;2:146–150
   • View In Article
4. Lawrence SJ, Botte MJ. Management of the adult, spastic, equinovarus foot deformity. Foot Ankle Int. 1994;15:340–346
   • View In Article
   • MEDLINE
5. Kottink AI, Oostendorp LJ, Buurke JH, Nene AV, Hermens HJ, Ijzerman MJ. The orthotic effect of functional electrical stimulation on the improvement of walking in stroke patients with a dropped foot: a systematic review. Artif Organs. 2004;28:577–586
   • View In Article
   • MEDLINE
   • CrossRef
6. Pittock SJ, Moore AP, Hardiman O, et al. A double-blind randomised placebo-controlled evaluation of three doses of botulinum toxin type A (Dysport) in the treatment of spastic equinovarus deformity after stroke. Cerebrovasc Dis. 2003;15:289–300
   • View In Article
   • MEDLINE
   • CrossRef
7. Lin SS, Sabharwal S, Bibbo C. Orthotic and bracing principles in neuromuscular foot and ankle problems. Foot Ankle Clin. 2000;5:235–264
   • View In Article
   • MEDLINE
8. Jaivin JS, Bishop JO, Braly WG, Tullos HS. Management of acquired adult dropfoot. Foot Ankle. 1992;13:98–104
   • View In Article
   • MEDLINE
9. Beckerman H, Becher J, Lankhorst GJ, Verbeek AL. Walking ability of stroke patients: efficacy of tibial nerve blocking and a polypropylene ankle-foot orthosis. Arch Phys Med Rehabil. 1996;77:1144–1151
   • View In Article
   • Abstract
   • Full-Text PDF (957 KB)
   • CrossRef
10. Trevena L, Barratt A. Integrated decision making: definitions for a new discipline. Patient Educ Couns. 2003;50:265–268
    • View In Article
    • Abstract
    • Full Text
    • Full-Text PDF (132 KB)
    • CrossRef
11. Frith L. Priority setting and evidence based purchasing. Health Care Anal. 1999;7:139–151
    • View In Article
    • MEDLINE
    • CrossRef
12. Hariharan S, Dey PK, Chen DR, Moseley HS, Kumar AY. Application of analytic hierarchy process for measuring and comparing the global performance of intensive care units. J Crit Care. 2005;20:117–124discussion 124-5
    • View In Article
    • Abstract
    • Full Text
    • Full-Text PDF (119 KB)
    • CrossRef
13. Cook DR, Staschak S, Green WT. Equitable allocation of livers for orthotopic transplantation: an application of the Analytic Hierarchy Process. Eur J Oper Res. 1990;48:49–56
    • View In Article
14. Castro F, Caccamo LP, Carter KJ, et al. Sequential test selection in the analysis of abdominal pain. Med Decis Making. 1996;16:178–183
    • View In Article
    • MEDLINE
    • CrossRef
15. Hummel JM, Snoek GJ, van Til JA, van Rossum W, IJzerman MJ. A multicriteria decision analysis of augmentative treatment of upper limbs in persons with tetraplegia. J Rehabil Res Dev. 2005;42:635–644
    • View In Article
    • MEDLINE
    • CrossRef
16. Saaty TL. How to make a decision: the analytic hierarchy process. Eur J Oper Res. 1990;48:9–26
    • View In Article
17. Dyer RF, Forman EH. Group decision support with the Analytic Hierarchy Process. Decis Support Syst. 1992;8:99–124
    • View In Article
18. Hatcher M. Voting and priorities in health care decision making, portrayed through a group decision support system, using analytic hierarchy process. J Med Syst. 1994;18:267–288
    • View In Article
    • MEDLINE
    • CrossRef
19. Dolan JG, Isselhardt BJ, Cappuccio JD. The analytic hierarchy process in medical decision making: a tutorial. Med Decis Making. 1989;9:40–50
    • View In Article
    • MEDLINE
    • CrossRef
20. Saaty TL. Decision-making with the AHP: why is the principal eigenvector necessary. Eur J Oper Res. 2003;145:85–91
    • View In Article
21. Dolan JG, Bordley DR. Using the analytic hierarchy process (AHP) to develop and disseminate guidelines. QRB Qual Rev Bull. 1992;18:440–447
    • View In Article
    • MEDLINE
22. Hummel JM, van Rossum W, Verkerke GJ, Rakhorst G. Medical technology assessment: the use of the analytic hierarchy process as a tool for multidisciplinary evaluation of medical devices. Int J Artif Organs. 2000;23:782–787
    • View In Article
    • MEDLINE
23. Kennedy A, Robinson A, Rogers A. Incorporating patients’ views and experiences of life with IBS in the development of an evidence based self-help guidebook. Patient Educ Couns. 2003;50:303–310
    • View In Article
    • Abstract
    • Full Text
    • Full-Text PDF (111 KB)
    • CrossRef
24. Robinson A, Thomson R. Variability in patient preferences for participating in medical decision making: implication for the use of decision support tools. Qual Health Care. 2001;10(Suppl 1):i34–i38
    • View In Article
25. Dolan JG, Bordley DR. Involving patients in complex decisions about their care: an approach using the analytic hierarchy process. J Gen Intern Med. 1993;8:204–209
    • View In Article
    • MEDLINE
    • CrossRef
26. Blankevoort L, Geertzen JH, Heerkens YF, Hijmans JM, Ursum J. Ontwikkeling van richtlijnen voor de indicatiestelling en het verstrekkingsproces van orthopedische orthesen (ORIVOO). Amsterdam: Eigen beheer; 2005;
    • View In Article