Volume 90, Issue 10 , Pages 1740-1748, October 2009
Evaluation of a Graphic Interface to Control A Robotic Grasping Arm: A Multicenter Study
Article Outline
- Abstract
- History of Rehabilitation Robotic Aids
- Difficulties Encountered With Rehabilitation Robots
- Objectives of the Study
- Methods
- Results
- Discussion
- Conclusions
- Acknowledgments
- References
- Copyright
Abstract
Laffont I, Biard N, Chalubert G, Delahoche L, Marhic B, Boyer FC, Leroux C. Evaluation of a graphic interface to control a robotic grasping arm: a multicenter study.
Objective
Grasping robots are still difficult to use for persons with disabilities because of inadequate human-machine interfaces (HMIs). Our purpose was to evaluate the efficacy of a graphic interface enhanced by a panoramic camera to detect out-of-view objects and control a commercialized robotic grasping arm.
Design
Multicenter, open-label trial.
Setting
Four French departments of physical and rehabilitation medicine.
Participants
Control subjects (N=24; mean age, 33y) and 20 severely impaired patients (mean age, 44y; 5 with muscular dystrophies, 13 with traumatic tetraplegia, and 2 others) completed the study. None of these patients was able to grasp a 50-cL bottle without the robot.
Interventions
Participants were asked to grasp 6 objects scattered around their wheelchair using the robotic arm. They were able to select the desired object through the graphic interface available on their computer screen.
Main Outcome Measures
Global success rate, time needed to select the object on the screen of the computer, number of clicks on the HMI, and satisfaction among users.
Results
We found a significantly lower success rate in patients (81.1% vs 88.7%; χ2 P=.017). The duration of the task was significantly higher in patients (71.6s vs 39.1s; P<.001). We set a cut-off for the maximum duration at 79 seconds, representing twice the amount of time needed by the control subjects to complete the task. In these conditions, the success rate for the impaired participants was 65% versus 85.4% for control subjects. The mean number of clicks necessary to select the object with the HMI was very close in both groups: patients used (mean ± SD) 7.99±6.07 clicks, whereas controls used 7.04±2.87 clicks. Considering the severity of patients' impairment, all these differences were considered tiny. Furthermore, a high satisfaction rate was reported for this population concerning the use of the graphic interface.
Conclusions
The graphic interface is of interest in controlling robotic arms for disabled people, with numerous potential applications in daily life.
Key Words: Arm, Muscular dystrophies, Quadriplegia, Rehabilitation, Robotics
List of Abbreviations: ANOVA, analysis of variance, HMI, human-machine interface, LIS, locked-in syndrome, MD, muscular dystrophy
PEOPLE WITH SEVERE disabilities, such as severe neuromuscular diseases, generally require supervision or assistance for most of their daily life tasks, resulting in significant expenses both for the health care system and patients' families.1 In the last 3 decades, advances in technology have greatly improved the independence and quality of life of persons with disabilities2, 3 with new electrically powered wheelchairs, HMI enabling computer control, modern orthotic devices, home disability equipment, and other electronic devices. Robotic arms designed to compensate for the lack of grasping ability in persons with disabilities are particularly interesting with very promising advances in robotics technology for both industrial and domestic applications.4, 5
The persons with disabilities who could benefit from using a grasping robot include patients with the following pathologies: MD, spinal cord injury, spinal muscular atrophy, multiple sclerosis, amyotrophic lateral sclerosis, cerebral palsy, rheumatoid arthritis, postpolio syndrome, LIS, and other severe motor paralysis. The number of people in the United States who could benefit from using this kind of robot is estimated at 150,000 at the most—that is, .06% of the population. In France, it is estimated that around 10,000 persons could benefit from robotic aids.6, 7
History of Rehabilitation Robotic Aids
Robotic rehabilitation began in the 1970s in several countries,8 usually stemming from new applications of industrial robots. European research and development in the field of rehabilitation robotics was established in the mid-1970s9 with the Spartacus and Heidelberg manipulator projects. The first robotic arms were included in stationary workstations10 like Raid in Europe, Master in France,9 or Devar in the United States.11 The robot was fixed to the floor or mounted on a rail to allow for horizontal arm movements. There were different end-effectors or grips, enabling performance of different types of tasks. Robot movements were preprogrammed and could be activated by pressing a button or selecting an icon on the computer. Examples of tasks that these robots could perform included picking up sheets of paper, turning the pages of a book, handling books, and inserting floppy disks into a computer or videotapes into a video cassette recorder. These workstations were too big to be used in a home environment and are now obsolete because all these tasks can be performed on a computer without the help of a robotic arm,12 such as e-mailing, online libraries, and listening to music.
The second concept of robotic aids included mobile robots consisting of compact and flexible arms mounted on the person's own wheelchair or on a mobile base. The most common directly controlled wheelchair-mounted robots are the Raptor13 and the Arm Unit Manus.14a Manus is a collaborative Dutch project started in 1984. It was built by Exact Dynamics in The Netherlands and has been widely distributed there for persons with muscular dystrophy. Manus is one of the most sold robots around the world. It consists of a robotic arm with 6 degrees of freedom, 80cm long, able to lift objects weighing up to 1.5kg with its terminal 2-fingered gripper. The user controls all the robot's movements with the help of a keypad or a joystick. With Manus, people can pick up objects on the floor, get things from shelves, open the refrigerator, pour themselves a glass of water, pick up a book, or open a door. The Raptor is a wheelchair-mounted American robot with 4 degrees of freedom that enables persons with disabilities to feed themselves and reach out to pick up objects from the floor, on a table, or above their heads. To our knowledge, this robot was not widely made available to persons with severe disabilities.
Besides the Manus and the Raptor, the most widely used domestic rehabilitation robot in the world is the Handy, which was developed by Topping in 198715 and enables people with poor control of their arms to complete daily life tasks independently such as eating, drinking, washing up, shaving, and brushing their teeth. The Middlesex University rehabilitation robot16 is an electrically powered wheelchair-mounted manipulator with a payload capacity higher than 1kg and a maximum reach of 0.7/0.9m. It has been tested on disabled and nondisabled users in the United Kingdom. To our knowledge, other devices—for example, the Italian robot URMAD (Mobile Robotic Unit for the Assistance to the Disabled) or the American ISAC robot—did not reach the stage of being validated by clinical trials.
Difficulties Encountered With Rehabilitation Robots
Despite the potential usefulness of wheelchair-mounted robotic arms,10, 17 at present only about 300 users worldwide benefit from various types of robots as assistive devices,18 with fewer than 10 users in France. Explanations for this phenomenon could be the lack of information, organizational and financial issues, psychologic issues, or technical difficulties with the devices themselves.19
Among technical difficulties, user experience shows that robots must meet certain requirements in terms of speed, easy-to-use functions, and security to enable a user to carry out the most common movements easily, quickly, and with minimal concentration. The way the robot is controlled by the user is essential, emphasizing the importance of HMI.
Until now, regarding the control of robotic arms by the operator, most researchers have focused on full control of the arm by the user at the time of task performance. Because it is necessary to control both position and orientation of the arm and closing/opening of the gripper, the main drawbacks of this type of control are that they require high concentration from the users and that it may take a long time for users to complete the task. The training period can also be quite lengthy—about 8 weeks for the Manus arm.19 As a result, this method is not practical for disabled patients with poor and limited control of hand movements.
Objectives of the Study
The goal of this study was to validate among users a recently developed HMI to control a robotic arm for persons with mild to severe disabilities. It consisted of a graphic interface improved with a panoramic sensor (omnidirectional vision sensor). The objective was to design the system, Aviso, to detect an object, grasp it, and bring it in front of the user automatically on receiving the activating command after the user selects a target on the computer screen.
Because the user's feedback is vital at all stages of the concept and design for equipment tailored to persons with disabilities, grasping functions have been assessed through user trials in 3 previous studies.
The first study involved 5 patients with quadriplegia who tested the first version of the device,20 leading to several technical enhancements: emergency stop, reliability of the grasping procedure, and writing of the presentation of the scene on the computer screen.
The second preliminary multicenter study took place in September 2005 and included 10 patients whose results were compared with those from 32 control subjects.21 Among the patients, there were 3 with muscular dystrophy, 3 with traumatic tetraplegia, 1 with LIS, 1 with amyotrophic lateral sclerosis, 1 with multiple sclerosis, and 1 with arthrogryposis. Through their usual human-computer interface, the testers used the graphic interface to control the robotic arm and grasp 5 different objects laid out within their visual fields. Results showed that this new interface was efficient in detecting objects in front of the users and that the satisfaction rate regarding ease-of-use and the overall training process was high among the group of patients. Some technical points requiring improvements and recommendations regarding some modifications were taken into account as suggested by the participants.
In a third unpublished study, to solve the problem of objects lying out of the users' visual field and to get ready for the design and set up of a mobile robotic arm that could grasp objects very far from the user, we suggested the use of a catadioptric vision sensor—the so-called panoramic camera—to help find these out-of-vision objects. The problem was first to define a way to transcribe a panoramic view of the environment on the computer screen. By involving 48 able-bodied participants (24 men; average age, 37y) in this last preliminary study, we found that the more comprehensible feedback for the end user was a 360° view displayed on the computer screen and centered by the subject, rather than 3 other previously tested presentations.
Based on the knowledge gained from these previous experiments, we conducted this multicenter study to validate our results on a larger group of patients, after technical enhancements resulting from the first evaluation, and to test the relevance of the panoramic camera to handle situations where the object is out of the embedded camera visual field. The objectives were (1) to validate the potential use of this new graphic interface in a multicenter sample of severely impaired patients, (2) to document the relevance of panoramic camera to detect objects out of the visual field, and (3) to evaluate users' satisfaction with this interface.
Methods
Device
The device consists of a Manus manipulator mounted on a static base. A catadioptric vision sensor, called the panoramic camera,b is fixed at the top of the robotic arm's base to give a global view of the mobile manipulator surrounding environment. A 360° image of the robotic arm's surrounding environment lets the user select the desired object displayed on the computer screen.
The Manus grip is equipped with 2 low-cost webcams, called embedded cameras, providing a local view of the surrounding environment. Information sent in from the webcams allows calculation of the arm motion needed to position the gripper just in front of the object. By triangulation, a 3-dimentional model representation of the object is computed in order to calculate the opening of the grip. An optical barrier inserted into the gripper's finger detects the presence of an object inside the gripper. Once the object has been located inside the fingers of the gripper, the arm grasps it and automatically brings it back in front of the user.
Safety considerations have also been taken into account, leading to emergency procedures so that the arm is kept collision-free from people: a red, easy-to-use push-button was installed next to the person evaluating the trial so the arm could be stopped if necessary. There is also a large icon on the computer screen that allows the user to stop the arm easily in case of an emergency.
Participants
We conducted a multicenter, open-label trial to evaluate the efficacy of the graphic interface Aviso enhanced by the panoramic camera to control the robotic arm. Patients were recruited from 4 physical medicine and rehabilitation units of French hospitals (Coubert, Reims, Berck sur Mer, Garches), all members of the French Association for the Promotion of New Technologies for Disabled People (Approche). Results obtained with the patients were compared with those reported for control subjects.
Inclusion criteria for patients were as follows: older than 18 years, with severe arm disability leading to the impossibility to grasp objects, and heavy lack of independence in daily life tasks. Exclusion criteria were as follows: patients were already involved in previous studies regarding the AVISO project, cognitive impairment, and inability to access a computer through any interface. Because of technical difficulties, people who needed a switch-scanning menu to control the computer could not be included.
To be included in the study, control subjects had to be older than 18 years and free from any neurologic or upper-limb orthopedic disease. Patient and control subjects were not appraised in age and sex.
Methods
The users sat in their wheelchair in front of a computer on a desk, with the robot on their left (fig 1). They could interact with the computer through their personal interface, the HMI.
Fig 1.
Installation of a patient in front of the desk with the robotic arm on his left.
Objects to grasp were set around the wheelchair and the robot (fig 2), not farther than 75cm from the central column of the Manus. Their locations and heights were selected according to functional considerations. In front of the patient and behind the screen of the computer, there was a table 63cm in height. The objects were chosen because they were easy to grasp, in order to test the AVISO graphic interface without the grasping complexities presented by smaller or bigger objects.
Fig 2.
Objects set around the user. The white circle indicates the Panoramic Camera.
Four of the 6 objects were laid out around this area: the soda can was directly set on the table, the pepper shaker was set on a support under the table (support height, 33.5cm), one of the water bottles was positioned on a support on the table (support height, 27cm), and a second one was placed on the floor (out of the participants' visual field). Two other water bottles were set at a distance of 53cm left of the robotic arm and could be within the users' visual field when they turned their heads to the left: 1 on the floor and 1 on another table (height, 77cm).
To select an object, the user had to first click on the object on the 360° panoramic view to orientate the embedded camera toward the object's direction. Using the panoramic camera was only optional: buttons on the HMI let the user move the gripper directly (translations along x-axis and z-axis, rotations around y-axis) and orientate the embedded camera. Panoramic camera use was also unnecessary when the object was already within the robotic arm's visual field. Once the object was in the visual field of the embedded camera, the user had to select it on the computer screen by drawing a box around it on the display. After selecting the object, the user clicked on the icon to start the automatic grasping procedure.
In each center, the evaluations lasted 1 week. They consisted in grasping 6 objects located in 6 different places all around the user and at different heights. Locations and heights of the objects were the same for all centers. A few minutes before starting the tests, each subject had a standardized 15-minute training period. Between each grasping procedure, the arm was brought up to a reference position so that the bottle on the floor behind the computer screen was not directly visible.
The procedure was as follows: (1) the therapist indicated the object to grasp; (2) the user was then free to choose whether to use the panoramic camera for correctly orientating the arm toward the object; (3) once the object was perfectly visible on the computer screen, the user drew a box around the object and launched the automatic grasping procedure. The users were asked to repeat each grasping task 3 times, thus amounting to 15 grasping tasks a user.
Evaluation Criteria
The primary evaluation criterion was the failure rate. This rate was calculated according to the total number of tasks that were not totally completed for each group. Failures caused by equipment dysfunctions were separated from user-related failure.
Secondary criteria were (1) task duration between the start signal and the start of the grasping procedure (ie, time necessary to correctly select the object on the screen), (2) percentage of sessions in which the panoramic camera was used by the user, (3) numbers of clicks necessary to complete the whole grasping procedure, and (4) user satisfaction evaluated through 3 questions on 4-level Likert scales. Furthermore, users' suggestions were noted and are briefly reported in the Results section.
Statistical Analysis
The success rate and user rate of the Panoramic Camera were evaluated with a chi-square analysis.
Normal parametric data were evaluated by repeated-measures ANOVA with the group (disability group, control group) as the between-subjects factor, the object/location (6 possibilities) and the trial number (first, second, third), as the within-subjects factors. An interaction between these 3 factors was looked for systematically. When ANOVA reported a statistically significant difference for the within-subjects factor, the Bonferroni correction test was used for pairwise comparisons. When an interaction with the between-subjects factor was observed, the ANOVA measure was repeated independently for the 2 groups.
Results
The main results are summarized in table 1.
Table 1. Main Quantitative Results of the Study
| Category | Global Rate of Success (%) | Rate of Success for Duration <79s (%) | Duration (s), Mean ± SD (range) | Use of the Panoramic Sensor (%) | No. of Clicks on HMI, Mean ± SD (range) |
|---|---|---|---|---|---|
| Patients | 81.1 | 65 | 71.6±49.1 (12–344) | 60.1 | 7.99±6.07(3–81) |
| Controls | 88.7 | 85.4 | 39.1±18.24(8–130) | 66.9 | 7.04±2.83(4–25) |
Participants
Evaluations involved a total of 44 participants: 24 control subjects (mean age, 33y; range, 19–55) and 20 disabled patients (mean age, 44y; range, 26–67). Among control subjects, there were 16 women and 8 men. Disabled individuals (7 women and 13 men) had muscular dystrophy or spinal muscular atrophy (n=5), traumatic tetraplegia (n=13), and Guillain-Barré syndrome (n=2). Among patients with traumatic tetraplegia, 4 had a neurologic level above C4 according to the international classification, 7 had a level between C5 and C6, and 2 had low tetraplegia. The 5 patients with hereditary diseases had severe impairment because the disease started in very early childhood. None of the 20 patients was able to grasp a 50cL full water bottle lying on a table in front of them.
Only 1 patient (number 35) and 1 control subject (number 11) knew the basic Manus device and had used it once. Control subjects used a mouse to interact with the computer. Disabled participants mostly used a trackball (n=12), some could use a simple mouse (n=6), and 2 needed head tracking. Most of them could select icons on the screen via a click on the mouse or on the trackball; 4 participants used an external contactor switch. Patients' characteristics are summarized in table 2.
Table 2. Patient Characteristics
| Patient No. | Age (y) | Sex | Laterality | Pathology | HMI: Displacement of the Pointer | HMI: Selection of the Icons | Center | HMI Experience |
|---|---|---|---|---|---|---|---|---|
| 1 | 65 | M | R | TT | TB | CM | G | 2 |
| 2 | 30 | M | R | MD | TB | SC | G | 3 |
| 3 | 50 | M | R | MD | HT | SC | G | 2 |
| 4 | 64 | F | R | GBS | TB | CM | G | 2 |
| 5 | 44 | M | R | TT | M | CM | G | 3 |
| 16 | 62 | F | R | TT | TB | CM | C | 1 |
| 17 | 44 | M | R | TT | TB | CM | C | 1 |
| 18 | 46 | F | R | TT | TB | CM | C | 1 |
| 19 | 67 | F | R | TT | TB | SC | C | 1 |
| 20 | 46 | M | R | TT | HT | SC | C | 3 |
| 26 | 53 | F | R | GBS | TB | CM | Re | 1 |
| 29 | 27 | M | L | SMA | Mo | CM | Re | 1 |
| 30 | 34 | F | R | MD | Mo(+feeders) | CM | Re | 1 |
| 33 | 26 | M | R | SMA | Mo | CM | Re | 1 |
| 34 | 43 | M | R | TT | Mo | CM | Re | 1 |
| 35 | 27 | M | R | TT | Mo | CM | B | 3 |
| 36 | 33 | F | R | TT | TB | CM | B | 1 |
| 37 | 33 | M | L | TT | TB | CM | B | 1 |
| 38 | 40 | M | L | TT | TB | CM | B | 1 |
| 44 | 45 | M | R | TT | TB | CM | B | 1 |
Success Rate
Control subjects showed a global success rate of 88.7% (49 of 432 tasks failed). Patients had a success rate of 81.1% (67 of 360 failures). The difference did not reach statistical significance (χ2, P=.017).
When considering all trials and all objects in both groups, 116 of 792 tasks failed. Among them, 92 resulted from robot or gripper malfunctions, or from failures of the visual servo control. Only 24 of 116 failures (16 of 67 among patients and 8 of 49 among control subjects)—that is, 20.7%—were related to the HMI itself: failure in locating the object on the 360° panoramic view, or failure in selecting the picture on the screen. Repartition between the 2 types of failures (users/equipment dysfunctions) was similar in both groups (χ2, P=.56).
The global success rate was not influenced by age, sex, or laterality of the participants, nor by the eventual use of the panoramic camera.
When considering the total number of successful tasks with a cut-off at 79 seconds for the allowed maximal task duration (ie, twice that of the mean time used by control participants to achieve the object selection on the screen—see the Task Duration Section below), a trial effect was observed in the patient group (ANOVA, P=.002). This result is described in figure 3.
Fig 3.
Percentage of success in both groups with a cut-off set at 79 seconds (ie, twice the mean duration necessary for the control subjects to complete the required task). The rate of success is plotted with the 3 successive trials (T1–T3) in patients (diagonal-lined bars) and controls (gray bars).
For impaired participants, the total number of successful tasks tended to improve with HMI experience, but this result did not reach significance (ANOVA, P=.094). The object's location also had a significant impact on the results (ANOVA, P=.001) because the water bottle lying on the floor in front of the user was harder to grasp than other objects. We did not find any pathology impact on this result.
Task Duration
Task duration was significantly higher for the impaired population (71.6s vs 39.1s; ANOVA, P<.001). Age did not influence task duration in either group.
In order to compare the 2 groups, we subsequently set a cut-off for the maximal duration time at 79 seconds, representing twice the amount of time needed by the control subjects to complete the task. In these conditions, the success rate for the impaired participants was 65% versus 85.4% for control subjects.
With a cut-off at 79 seconds, the trial number significantly influenced the task duration, both in patients and in the control group (ANOVA, group effect, P<.001; trial effect, P<.001; interaction, P=.72). The impact of the object's location was also significant in both groups (ANOVA object impact, P<.001; interaction group-object, P=.303). Figure 4 illustrates these 2 results.
Fig 4.
Impact of the trial number (T1, T2, T3) and of the object to be grasped (Ob1–Ob6) on the duration of the task selection in both groups.
The task duration was longer for objects for which the panoramic camera was used—that is, the 2 objects on the left and the out-of-view bottle on the floor (ANOVA, P=.001). When using the Panoramic Camera, participants needed to select more icons on the screen, and the device needed time to orientate the embedded camera toward the object. Thus, extending the selection time did not necessarily translate into enhanced difficulties in completing the task but rather was correlated to technical issues.
Panoramic Camera Use
The panoramic camera was used in 66.9% of the successfully completed tasks for control participants and in 60.1% for disabled users. It was specifically chosen for the objects that the subjects could not see, on the left side of the table or on the floor (fig 5).
Fig 5.
Panoramic Camera percentage of use in patients (left) and control subjects (right).
Panoramic Camera use was not influenced by the trial number (ANOVA, trial effect, P=.205; interaction group-trial, P=.468). Nevertheless, we noted that for the control subjects, the experience, as reflected by the trial number, showed a decreasing tendency for panoramic camera use that was not observed for the disabled participants group (fig 6).
Fig 6.
Percentage of use of the Panoramic Camera among patients (left) and control subjects (right). Impact of experience illustrated by the results in the 3 successive trials.
Numbers of Clicks Necessary to Complete the Whole Grasping Procedure
The mean ± SD number of clicks necessary to complete the whole grasping procedure was 7.97±6.07 in patients versus 7.04±2.87 in controls. The difference reached statistical significance (ANOVA, P = .007). A total of 7 clicks is easy to explain. Correctly positioning the arm before grasping the object needed about 4 clicks depending on its location. Then, the object definition and the automatic grasping required 3 clicks: 2 for drawing the box around the picture, 1 to start the visual servo.
Satisfaction on 4-Level Likert Scales
Evaluations of the Aviso users' satisfaction were very positive:
Qualitative Evaluations Based on Questionnaires
Most of the subjects (42/44) considered the system very easy to use and answered that the objects' selection was made easier thanks to the panoramic camera. On the other hand, the panoramic camera visual feedback was deemed too blurry by 29 of 44 of the participants, and its use was disturbed by blind angles for 19 of 44 of them.
Concerning the potential use of the robotic arm, 18 of 20 patients would prefer to use Manus on a mobile platform rather than mounted on the wheelchair.
Discussion
Clinical Evaluation of Robotic Technical Aids
In the field of technical aids, the fact that users (or patients) and medical staff are included at the beginning of the development of devices is essential, because the real needs and expectations of disabled people are taken into consideration at the initial stage of the process. Basic evaluations of finished products usually lead to marketing mistakes and failures, rendering these devices useless or inadequate.3, 18, 22, 23
Graphic Interface to Control Grasping Robots
HMI to control robots is one of the most important technical points requiring careful attention. Currently, a user may manually control the arm unit Manus by accessing menus via standard access devices, such as a keyboard, joystick, or single switch. The joint menu mode allows the user to manipulate the Manus arm by moving its joints individually. The Cartesian menu mode lets the user move the gripper linearly through the 3-dimensional x-y-z plane. In any case, this Manus control is not intuitive because it requires a high level of cognitive awareness.24 In addition to manual control, the Manus arm can be controlled by communication from a personal computer and is therefore programmable.
Previous studies mentioned other classic HMIs that can be used for moving a robotic arm: breath or head motion,25 for example. To our knowledge, none have reached clinical validation or are available on the market.
Graphic interface has rarely been proposed to control robots for disabled patients. Tsui et al26 built a method applied to Manus control in which the objects' selection was made possible through a graphic interface compatible with single-switch scanning. Two cameras were mounted on the robotic arm: one on its shoulder and the other one on its gripper. With this method, the user is presented with an interactive image of the shoulder view, divided into 4 quadrants. When the quadrant containing most of the objects the user wishes to manipulate is highlighted, the user turns on the switch to select it. The quartering procedure is then repeated a second time, providing a view that is one sixteenth of the original image area. The Manus assistive robotic manipulator then moves in the x-y plane toward the center of the selected quadrant, emulating human motion control. No real grasping procedure was included in this study, but people were asked to move the gripper as close as possible to the selected object. The authors tested their device on 12 able-bodied participants, comparing moving the arm with its usual interface and moving it through the graphic interface. In the exit questionnaire, 83% of the users preferred the robot's manual control rather than using the graphic interface. The system workload, as defined by the total number of clicks on the interface needed to complete the task, was significantly higher with manual control. The precision of the gripper end position next to the object was significantly greater with manual control. These preliminary results are not convincing. Two elements could explain these poor results: tests were conducted with an unfinished device, and experiments did not include disabled patients who could really benefit from it.
Our study shows very promising results regarding this kind of interface. The very high rate of success in grasping objects with Aviso, the fact that most of the failures reported in this study are a result of technical failures and not human ones, and the high satisfaction rate among patients underline the high relevance of the graphic interface to pilot a robotic arm for persons with disabilities. Its easy-to-use design is underlined by the very short duration time needed to grasp an object, because 75% of the tasks performed by patients lasted about twice as long as the time required by control subjects to complete the task, and by the very low number of clicks necessary to select the object on the screen.
Our results emphasize the importance of the learning process in adapting to this type of device, because the trial number (1, 2, 3) often had an impact on the results. We also validated through this work the importance of the interface to control the computer, which seems to be one of the major restrictions for using the graphic interface.
The very high diversity of the pathologies encountered by patients included in our study could explain the variance in some results (task duration, rate of success, and so forth). Control subjects were younger than patients, and this point could reduce the difference between the 2 populations: older persons are supposed to be less adaptable to technical aids, particularly to computers. For us, this point emphasizes the significance of the results.
Visual Sensors and Panoramic Cameras to Control Robotic Aids
The question regarding the best way to localize objects to be grasped in the user's surrounding environment remains unclear. Previous studies have reported methods that can be subjected to criticism. For example, Versluis et al27 presented a rehabilitation robotic system in which the object was selected using a pointer (laser beam). This object detection system may not be applicable for persons with poor or limited control of their hands. Other methods rely on using a marker on the object in order to detect it more easily.28 This method can only work for known and labeled objects and is not very useful in daily life tasks.
Vision sensors have already been proposed by other authors to select objects in the surrounding environment of the gripper. Farahmand et al29 presented an automatic robot arm equipped with stereovision detection in order to distinguish different objects from the background. After detecting the target, the system moves the robot arm toward the object, grasps it, moves it to a predefined position in front of the user, releases it, and finally returns to its home position. The system is automatic and is operated by only 1 command. Unfortunately, no clinical evaluation of the device is available.
In a recent study, Saxena et al30 presented a learning algorithm that was able to identify grasp locations directly from images of new objects and demonstrated this approach on 2 robotic manipulation platforms. In this study, the camera mounted onto the arm gripper enabled the robotic arm to grasp a wide range of unknown objects.
Finally, visual sensors can be coupled with other sensors to render the object recognition and the following tracking procedure easier. Martens et al28 coupled a camera with an intelligent tray to make the localization easier. In the Care-o-bot II project,31 the camera was coupled with a laser scanner to associate 3-dimensional information with the object's appearance.
Most systems suppose that the object is already inside the camera's visual field, which is not the case when the robot is used in real-life conditions. Other studies offer solutions to handle situations in which the object is out of the gripper-embedded camera's visual field. Dune et al32 described a wide-angle camera assisting an embedded camera mounted onto a robotic arm by observing the whole scene. Knowing the respective position of the 2 sensors, the object identified by 1 click on the remote's fixed view was then automatically searched with the embedded camera. This is a first step to investigate the possible partnership between 2 or more cameras in prehension tasks. Kragic et al33 proposed use of 2 sets of stereo rig: a peripheral one, with a wide visual field to locate the potential object to be grasped, and a foveal one, giving a more focused local description. To our knowledge, these are laboratories studies, and these methods have never been applied to or assessed for persons with disabilities.
In our study, the spontaneous use of the panoramic camera by patients is surprisingly less important than in control subjects. Before the study, we thought that impaired persons would use the panoramic camera much more than control subjects because of the supposed enhanced function of the interface when a 360° view is available. This result suggests that the panoramic camera induces an increased complexity in using this interface, probably because people are not familiar with a panoramic view of their surroundings. As suggested from the questionnaire results, users do not find this panoramic camera very useful, but this does not mean that it is useless, because out-of-view objects could not be grasped without this function. Integrating panoramic camera manipulation might be quite difficult when the cognitive load induced by the task itself is already quite heavy. This hypothesis is backed up by the reported poor panoramic camera use improvement rate by patients after 3 trials. Similar findings have already been reported in other studies with other technical aids,34 underlining the idea that increasing the cognitive load for using high-technology devices should be taken into account, because it requires a longer training period before use, and paradoxically there is a potential difficulty in using the device.
Several improvements for this 2-camera device are being looked at. We would like to implement automatic image matching between the 2 vision sensors (the embedded camera and the panoramic camera) in order to get an automatic object location and grasping after clicking on the screen. Furthermore, we plan to install a picture library of common objects into the computer to facilitate the finding of objects. Natural speaking could be the most intuitive interface to pilot robots in the future.
Intelligent Robots
Intelligent robots, which incorporate artificial intelligence in their controlling software, are the next step in bringing the rehabilitation robots up to their full potential. In this area, the HMI improvement is essential, as are integration of automatic motion planning, enhancement of computing resources for environment recognition, and development of adaptive behaviors enabling the robot to interact efficiently with humans.35
For example, H75 is able to perform an object-tracking task based on the object's color, the object's most recently detected position, or a search program drawing on the a priori known shape of the object and the stereo cameras mounted on its head.
Currently, the automatic tasks performed by our device are limited to grasping followed by bringing the object to the user. However, once the object is grasped and brought to the users they are currently able to control Manus with the old interface in order to execute a nonautomatic task, like filling up a glass with the grasped bottle of water. We believe that in the future, we will be able to increase and diversify the different automatic tasks.
Future of the Field of Robotic Grasping Aids for Disabled People
Further improvements in robotic technologiesWhether a person wants to use a robotic aid depends on several factors and not only on what the robot can be used for, how easy it is to use, and what it looks like. It also depends on the user's priorities and on the availability of other assistive devices. Some persons may prefer to be assisted by a personal assistant, relative, or animal instead of a robot. The acceptance of electronic aids in general is supposed to be quite widespread in this disabled population.2
Among important technical problems encountered by the users is the structure of the robot arm: robots should be small, light, silent, and sober in design. The size and weight of the wheelchair-mounted Manus manipulator may have a negative impact on accessibility and driving precision.
Mobile and independent robotsMobile robots represent an attractive solution that could avoid some of the aforementioned pitfalls. This theory is supported by the responses obtained to our open questionnaire. In rehabilitation, they were first described in the 1990s by Regalbuto et al,25 who built a mobile robot made of a Hero 2000 manipulator arm mounted on a mobile platform and controlled via a remote station and a wireless communications link. Other authors mentioned mobile robots as very promising devices.36
Integration of the robot with other assistive devicesFor users with severe disabilities, it may be beneficial to have an integrated control system in which a single-control interface is used to operate 2 or more assistive devices25, 37: power wheelchairs, increased communication devices, computers, environmental control units, robots, and other devices that are electronically controlled. The advantages are mainly that the single access site allows for controlling several different devices without any assistance and that the user does not need to learn a different operating system for each device. On the other hand, using more than 1 device through the same integrated control system increases safety concerns: for example, if the power source goes down, the user is left with no systems to use. Accordingly, more studies are necessary to ensure safety risks are not increased.
Broadening of indications for robotics applications in medical careThis work focused on robots as compensatory assistive devices for people with severe disabilities. Future applications of robotic technology will continue to provide advances in rehabilitation, particularly as augmentation to the rehabilitation process. The first studies were done in the 1990s, focusing on patients with stroke.38 During the 10 last years, about 5 robotic devices have been assessed in clinical controlled trials and are now used in current care39: the Mit-Manus and its commercial version the In Motion Shoulder Elbow Robot, the Assisted Rehabilitation and Measurement Guide (Arm Guide), and the Mirror Image Motion Enabler.
Conclusions
This study demonstrates that graphic interface may be of interest in controlling robotic arms for disabled people. It is very promising concerning future applications of robots as efficient technical aids that could ameliorate the quality of life of severely impaired patients. Nevertheless, this method is not in itself sufficient, and it may be improved by integrating other advances in robotics technology and computational sciences to render the human-machine interface easier for the most disabled patients.
An increasing number of laboratories and clinical teams are concerned with studies for the medical applications of robotic devices. This phenomenon leads to quick technical enhancements and exponential widening of their possible applications.
The potential number of users for such devices is increasing: strongly impaired patients who cannot use available robots because of too severe disabilities (eg, LIS, adults with MD), patients with cognitive impairment who need a very intuitive interface to control technical aids, and elderly people, a population that is greatly increasing throughout the world.
Suppliers
Acknowledgments
This study was conducted in partnership with the Approche association and its members who were not listed as authors. Approche is the French Association for the Promotion of New Technologies for Disabled People.
References
- . Mapping the journey: family carer's perceptions of issues related to end-stage care of individuals with muscular dystrophy or motor neurone disease. J Palliat Care. 2003;19:36–42
- . Psychosocial impact of electronic aids to daily living. Assist Technol. 2000;12:123–131
- . Satisfaction with technological equipment in individuals with tetraplegia following spinal cord injury. Ann Readapt Med Phys. 2007;50:78–84
- Robomaut: a robot designed to work with humans in space. Auton Robots. 2003;14:179–197
- . The experimental humanoid robot H7: a research platform for autonomous behaviour. Philos Transac A Math Phys Eng Sci. 2007;365:79–107
- Cost-effectiveness of the Manus robot manipulator. In: Mokhtari M editors. Integration of assistive technology in the information age (Vol 9, Assistive Technology Research Series). Amsterdam: IOS Pr; 2001;p. 340–345
- . Processes for obtaining a “Manus” (ARM) robot within The Netherlands. Adv Rehabil Robot. 2004;306:221–230
- . The Spartacus telethesis: manipulator control studies. Bull Prosthet Res. 1979;16:69–105
- . The robotized workstation “MASTER” for users with tetraplegia: description and evaluation. J Rehabil Res Dev. 1999;36:217–229
- . Clinical experience in rehabilitation robotics. J Biomed Eng. 1991;13:239–243
- . Workstation robotics: a pilot study of a Desktop Vocational Assistant Robot. Am J Occup Ther. 1993;47:1009–1013
- . Evaluation of a robotized MASTER/RAID workstation at home: protocol and first results. In: Mokhtari M editors. Integration of assistive technology in the information age (Vol 9, Assistive Technology Research Series). Amsterdam: IOS Pr; 2001;p. 299–306
- . The raptor wheelchair robot system. In: Mokhtari M editors. Integration of assistive technology in the information age (Vol 9, Assistive Technology Research Series). Amsterdam: IOS Pr; 2001;p. 135–141
- . MANUS—a wheelchair-mounted rehabilitation robot. Proct Inst Mech Eng [H]. 2001;215:285–290
- . An overview of the development of Handy 1, a rehabilitation robot to assist the severely disabled. J of Intelligent Robot Syst. 2002;34:253–263
- . The Middlesex University rehabilitation robot. J Med Eng Technol. 2005;29:151–162
- . An electric wheelchair mounted robotic arm—a survey of potential users. J Med Eng Technol. 1990;14:143–154
- . Assessing human-robotic performance for vocational placement. IEEE Trans Rehabil Eng. 2000;8:394–404
- . MANUS towards a new decade. In: Mokhtari M editors. Integration of assistive technology in the information age (Vol 9, Assistive Technology Research Series). Amsterdam: IOS Pr; 2001;p. 155–161
- Intelligent interface for robot object grasping (Handicap 2006). In: Barcelona, Spain: Advancement of Modelling and Simulation Techniques in Enterprises (AMSE) Periodicals; 2006;p. 106–1162006 (67 Suppl), p 106-16.
- Robot grasping of unknown objects, description and validation of the function with quadriplegic people. In: Proceedings of ICORR 2007. IEEE 10th International Conference on Rehabilitation Robotics. IEEE Pr; 2007;p. 35–42
- . Assessment of assistive technology in physical medicine and rehabilitation. In: Barat M, Franchignoni F editor. Assessment in physical medicine and rehabilitation: views and perspectives. Pavia: Maugeri Foundation Books; 2004;p. 195–203
- Un CITH spécialisé dans la technologie pour le handicap moteur: quatre ans d'expérience. Ingenierie et Recheche BioMédicale (IRBM). 2006;27:174–178
- . Integrated control of Manus manipulator and wheelchair enhanced by environmental docking. Robotica. 1998;16:491–498
- . Toward a practical mobile robotic aid system for people with severe physical disabilities. J Rehabil Res Dev. 1992;29:19–26
- . Development and evaluation of a flexible interface for a wheelchair mounted robotic arm. In: Proceedings of the 3rd ACM/IEEE International Conference on Human Robot Interaction. Amsterdam: ACM; 2008;p. 105–112
- . Enhancing the usability of the Manus Manipulator by using visual servoing. In: Proceedings of ICORR 2003. International Conference on Rehabilitation Robotics. Daejon, South Korea: IEEE Pr; 2003;p. 43–46
- Martens C, Ruchel N, Lang O, Ivlev O, Graser A. A friend for assisting handicapped people. IEEE Robotics and Automation Magazine 8, 1 (2001) IEEE Pr; 2003. p 57-65.
- . An intelligent assistive robotic manipulator. Conf Proc IEEE Eng Med Biol Soc. 2005;5:5028–5031
- . Robotic grasping of novel objects using vision. Inter J Robot Res. 2008;27:157–173
- . Care-o-bot II development of a next generation robotic home assistant. Autonomous Robots. 2004;16:193–205
- . One click focus with eye-in-hand/eye-to-hand cooperation. In: Proceedings of the International Conference on Robotics and Automation (ICRA) 2007. IEEE Pr; 2007;p. 2471–2476
- . Vision for robotic object manipulation in domestic settings. Rob Auton Syst. 2005;52:85–100
- Home trials of a speech synthesizer in severe dysarthria: use patterns, satisfaction and utility of word prediction. J Rehabil Med. 2007;39:399–404
- . How to make an autonomous robot as a partner with humans: design approach versus emergent approach. Philos Transact A Math Phys Eng Sci. 2007;365:21–47
- . Localization and control of a rehabilitation mobile robot by close human-machine cooperation. IEEE Trans Neural Syst Rehabil Eng. 2001;9:181–190
- Integrated control and related technology of assistive devices. Assist Technol. 2003;15:89–97
- . Robot training enhanced motor outcome in patients with stroke maintained over 3 years. Neurology. 1999;53:1874–1876
- . Effects of robot-assisted therapy on upper limb recovery after stroke: a systematic review. Neurorehabil Neural Repair. 2008;22:111–121
Supported by the Fondation Caisses d'Epargnes pour la Solidarité.
No commercial party having a direct financial interest in the results of the research supporting this article has or will confer a benefit on the authors or on any organization with which the authors are associated.
PII: S0003-9993(09)00408-0
doi:10.1016/j.apmr.2009.05.009
© 2009 American Congress of Rehabilitation Medicine. Published by Elsevier Inc. All rights reserved.
Volume 90, Issue 10 , Pages 1740-1748, October 2009
