A Framework for Measuring the Progress in Exoskeleton Skills in People with Complete Spinal Cord Injury

For safe application of exoskeletons in people with spinal cord injury at home or in the community, it is required to have completed an exoskeleton training in which users learn to perform basic and advanced skills. So far, a framework to test exoskeleton skills is lacking. The aim of this study was to develop and test the hierarchy and reliability of a framework for measuring the progress in the ability to perform basic and advanced skills. Twelve participants with paraplegia were given twenty-four training sessions in 8 weeks with the Rewalk-exoskeleton. During the 2nd, 4th, and 6th training week the Intermediate-skills-test was performed consisting of 27 skills, measured in an hierarchical order of difficulty, until two skills were not achieved. When participants could walk independently, the Final-skills-test, consisting of 20 skills, was performed in the last training session. Each skill was performed at least two times with a maximum of three attempts. As a reliability measure the consistency was used, which was the number of skills performed the same in the first two attempts relative to the total number. Ten participants completed the training program. Their number of achieved intermediate skills was significantly different between the measurements X_F²(2) = 12.36, p = 0.001. Post-hoc analysis revealed a significant increase in the median achieved intermediate skills from 4 [1–7] at the first to 10.5 [5–26] at the third Intermediate-skills-test. The rate of participants who achieved the intermediate skills decreased and the coefficient of reproducibility was 0.98. Eight participants met the criteria to perform the Final-skills-test. Their median number of successfully performed final skills was 16.5 [13–20] and 17 [14–19] skills in the first and second time. The overall consistency of >70% was achieved in the Intermediate-skills-test (73%) and the Final-skills-test (81%). Eight out of twelve participants experienced skin damage during the training, in four participants this resulted in missed training sessions. The framework proposed in this study measured the progress in performing basic and advanced exoskeleton skills during a training program. The hierarchical ordered skills-test could discriminate across participants' skill-level and the overall consistency was considered acceptable.

Introduction

Worldwide the incidence of Spinal Cord Injury (SCI) is 180,000 cases per annum (Lee et al., 2014) of whom 50% have a complete lesion and become wheelchair-designated for their mobility (Wyndaele and Wyndaele, 2006). A lifetime of sitting has been associated with an increased risk of multiple secondary complications, such as pressure ulcers, spasticity, and worsening of bladder and bowel dysfunction (Jensen et al., 2013; Adriaansen et al., 2016). Exoskeletons (external active orthosis) make it possible for people with paraplegia to regain their standing and walking mobility by generating the basic motions for ambulation e.g., standing-up, sitting-down, standing, and walking. Similar to other standing and robotic gait training devices (Middleton et al., 1997; Dunn et al., 1998; Mirbagheri et al., 2015), exoskeletons have the potential to prevent secondary health complication (Miller et al., 2016). The main benefit of exoskeletons compared to other robotic gait training devices (such as Lokomat^®) is that exoskeletons can be used at home and in the community outside of a clinical setting (Federici et al., 2015). However, several risks are identified with exoskeleton use such as falls, joint misalignment, skin damage, software malfunctions, electrical and fire hazard, and user errors (He et al., 2017). So far, the chance and extent of the risks are not well understood (He et al., 2017). Furthermore, manufacturers require an intensive training period before home and community use is allowed.

A prerequisite for safe and independent home and community exoskeleton use, is that users are able to perform basic and advanced exoskeleton skills. Previous research mainly focussed on the basic skills (sit-to-stand, stand-to-sit, and walking) and has shown that basic skills can be learned in a 25 sessions-training program with varying levels of assistance (Spungen et al., 2013; Kozlowski et al., 2015; Platz et al., 2016). The basic skills are highly relevant for use in a clinical setting, but for safe independent community use more advanced skills are required, including arresting gait on command, passing door thresholds, low curbs and ramps and controlling the input device (Spungen et al., 2013; Yang et al., 2015). The control of and interaction with the exoskeleton is diverse across the different exoskeletons available on the market. Moreover, some exoskeletons are more difficult or impossible to control dependent on the level and severity of the SCI of the user (Bryce et al., 2015). Several studies tested advanced skills in a limited number of motor complete SCI patients (Spungen et al., 2013; Hartigan et al., 2015; Kozlowski et al., 2015; Benson et al., 2016; Platz et al., 2016). However, the advanced skills were not tested in a systematic way and for example Spungen et al. concluded that the skills could have been introduced earlier in the training program (Spungen et al., 2013). So far, a systematic framework to structure, test and evaluate exoskeleton skills during a training program is lacking. Therefore, the aim of this study is to develop a framework to measure the progress in the ability to perform exoskeleton skills. The proposed framework consists of exoskeleton skills arranged into a hierarchy so that the difficulty increased with each tested skill. If the exoskeleton skills formed a true hierarchy and a skill was not achieved, it can be assumed that the participant would not achieve all higher skills and would achieve all lower skills. Therefore, arranging the skills into a hierarchy would reduce the time and effort of the exoskeleton-skills-test (Tyson and DeSouza, 2004). Furthermore, it is essential that the exoskeleton-skill-tests in the framework are reliable. Accordingly, the skills had to be performed consistent to reduce the change of misjudging the participants' skill-level.

Before an advanced exoskeleton skill-level can be achieved, an intensive training program with multiple training sessions per week over a longer period of time is required. The risk factors associated with such an intensive training program are still not well understood (He et al., 2017). However, it can be expected that such an intensive training program decreases the risk of falls, joint misalignment and user errors and increases the safety of exoskeleton home and community use. On the contrary, intensive exoskeleton use increases the risk of skin damage and bruises. Previous research concluded that in hospital training with an exoskeleton was safe (Zeilig et al., 2012; Esquenazi, 2013; Kolakowsky-Hayner et al., 2013; Kubota et al., 2013; Spungen et al., 2013). However, other studies disclosed mild to moderate skin damages in half of the participants (five out of ten) (Benson et al., 2016) (four out of seven) (Platz et al., 2016). Other reported complications were a fracture of the talus (Louie et al., 2015) and venous-lymphatic stasis in the lower limbs (Onose et al., 2016). Hence, assessing the occurrence of complications such as skin damage, muscle or joint pain, incontinence problems, device related errors, fractures, venous-lymphatic stasis, and falls during an exoskeleton training program is important for clinical recommendations.

In conclusion, the main objective of this study was to develop and test a framework for measuring the progress to perform basic and advanced exoskeleton skills in a group of individuals with motor complete SCI. The hierarchy and the reliability of the exoskeleton-skills-test in the framework was evaluated. As a secondary outcome, complications such as skin damage, muscle or joint pain, and incontinence problems resulting from the intensive exoskeleton training program were assessed.

Materials and Methods

Participants

People with paraplegia who gained knowledge about the exoskeleton technology throughout the media and who were interested in testing the potential of an exoskeleton contacted the rehabilitation center of the Sint Maartenskliniek to participate in this study. Eligible persons were adult patients in the chronic phase (>6 months) with a motor complete SCI [American Spinal Injury Association Impairment Scale (AIS) A or B] between Thoracic 1 (Th1) and Lumbar 1 (L1). The exclusion criteria were physical factors that hamper proper functioning of the exoskeleton, such as severe spasticity (Modified Ashworth Scale > 3), taller than 1.90 m or smaller than 1.60 m, bodyweight above 100 kg, and restricted range of motion in the hip, knee, or ankle joint. Other exclusion criteria were inability to control crutches, unable to make a transfer from a chair to a wheelchair without the use of external support, and patients with conditions that could interfere with the motor learning process (e.g., stroke). Potential subjects with an increased risk of adverse events such as patients with osteoporosis, fractures of the lower extremities in the last 2 years, balance disorders, neurogenic heterotopic ossification and pregnancy were also excluded. All participants gave written informed consent in accordance with the Declaration of Helsinki. The study was approved by the medical ethics committee of Arnhem-Nijmegen (2016-2418) and the internal review committee of the Sint Maartenskliniek.

Procedure

All exoskeleton training sessions and measurements were performed in the sports hall at the rehabilitation center. Prior to the start of the training a brief physical examination by a rehabilitation physician was performed, in which the in- and exclusion criteria of the study were checked. Participants were given twenty-four training sessions of 1.5-h over an 8 week period. Three physical therapists were trained by ReWalk™ Robotics to give the exoskeleton training. During each session at least two physical therapists were present to assure safety. The exoskeleton and the Lofstrand crutches were adjusted to the patients' body composition during the first training session. After each training the physical therapists notated the skills that were practiced. Participants kept a logbook during the entire study including any complications such as skin abrasions, muscle or joint pain, falls, and incontinence problems. The logbook was filled out at least three times a week. To assess the progress in achieved skills the participants' skill-level was tested every 2 weeks during a training session. In total the skill-level was assessed four times during the study, three times with the Intermediate-skills-test and one time with the Final-skills-test.

Intermediate- and Final-Skills-Test

The Intermediate-skills-test was performed in training week 2, 4, and 6. The Intermediate-skills-test consisted of 27 skills, which were measured separately of each other and were arranged into a hierarchy so that the difficulty increased with each skill. The intermediate skills were sorted into three categories; standing, walking, and advanced skills. Each subsequent category required more control of the user over the exoskeleton. Within each category the complexity of the skills also increased. In the standing skills the feet of the user remained roughly at the same place and participants learned to use their crutches, whereas there was displacement of the feet in the walking and advanced skills. In the walking skills, the increase of difficulty was related to the decrease in level of assistance and number of involuntary stops. In the advanced skills an additional task was performed while walking. The complexity of the task increased from walking turns with a decrease in number of involuntary stops to passing obstacles to passing obstacles that require raising or lowering of the center of mass (walk up and down a martial arts mat). An overview of the 27 skills of the Intermediate-skills-test is given in Table 1. Each intermediate skill was performed at least two times with a maximum of three attempts. An intermediate skill was considered achieved when the skill was performed independent without assistance of the exoskeleton trainer in at least two out of three attempts. The Intermediate-skills-test continued until two skills were not achieved. Participants were allowed to take rest between the various skills tested. A more detailed description of the Intermediate-skills-test is provided in Supplementary Table 1 (available online).

TABLE 1

Table 1. Assessed exoskeleton skills in the Intermediate-skills-test.

The Final-skills-test was performed during the last training week (week 8) in the final training session. A prerequisite for performing the Final-skills-test was that participants could control the remote control and walk without assistance of the exoskeleton trainer. The Final-skills-test consisted of a fixed set of 20 skills and was performed two times with a 5 min break in between. In contrast to the Intermediate-skills-test, the tested exoskeleton skills were measured in sequence during the Final-skills-test, simulating daily life situations in which skills are rarely performed independent of each other. Moreover, performing skills in sequence made it more difficult to achieve a skill than performing skills independent from each other (e.g., arresting gait immediately after a sharp curve compared to arresting gait independent of the previous action). In the Final-skills-test the focus was on independent performance of skills and the number of stops was not taken into account. Furthermore, the basic intermediate skills (e.g., weight shifting, touching the wristband, sit-to-stand, and assisted walking) are required in performing most of the skills and were not tested separately. In order to assess the test in a sports hall with as little material as possible, the advanced intermediate skill of opening a door was not part of the Final-skills-test. To assure safety, the exoskeleton trainer walked behind the participant but did not intervene unless the participants lost their balance and could fall. The final skills were considered achieved when the participants performed the skills without assistance of the exoskeleton trainer. In Figure 1 a schematic representation of the Final-skills-test is given.

FIGURE 1

Figure 1. Schematic representation of the top view of the Final-skills-test. Arrows represent the walking direction.

Equipment

In this study, two wearable robotic exoskeletons that enable powered hip and knee motion from ReWalk™ Robotics were used; (1) the ReWalk™ Rehabilitation System and (2) the ReWalk™ Personal 6.0. The exoskeleton systems provided user-initiated mobility through the integration of a wearable brace support, a computer-based control system and motion sensors. The exoskeleton systems have the Class II FDA clearance for both use in a rehabilitation setting as well as personal use. All participants started training with the ReWalk™ Rehabilitation System. Only participants who met de criteria to perform the Final-skills-test used the ReWalk™ Personal 6.0 system as well.

Data and Statistical Analysis

To assess if the proposed framework measured the ability to perform basic and advanced exoskeleton skills throughout an exoskeleton program, the skill-tests in the framework should measure progression in the number of achieved skills and show distinct skill-levels between participants. The skills should be arranged into a hierarchical order of difficulty. Moreover, the skills tested in the framework should be performed consistent. In addition, the relation between the Intermediate- and Final-skills-test was determined.

Achieved Intermediate Skills

The number of achieved skills was analyzed using descriptive statistics (median and ranges). Differences in the number of achieved skills between the three Intermediate-skills-test was assessed with the non-parametric Friedman test (α = 0.05). In case of a significant Friedman test, Wilcoxon post-hoc test with Bonferroni correction (α = 0.017) was used to determine changes. The number of participants who showed the expected increase in number of achieved skills over the three intermediate measurements was determined. Each intermediate skill was also analyzed separately for the number of times a skill was achieved.

Hierarchy of the Skills

The hierarchy of the skills tested in the Intermediate-skills-test was analyzed according to two measurements (1) the rate of participants achieving each intermediate skill and (2) the coefficient of reproducibility (Tyson and DeSouza, 2004). Both tests are based on the theoretical expectation that the participants ability to achieve a skill would decrease as the difficulty of the task increased. For a more detailed description see Tyson and DeSouza (2004). The coefficient of reproducibility was calculated with the formula described by Tyson and DeSouza: Coefficient of reproducibility = 1 − scaling errors/(number of skills × number of observations; Tyson and DeSouza, 2004). In which scaling errors is the number of participants who did not achieve the skills in the predetermined order. Since participants progressed during the training each intermediate skills measurement was considered as a separate observation in the analysis. A coefficient of reproducibility of at least 0.9 was considered acceptable (Guttman, 1944; Tyson and DeSouza, 2004).

Achieved Final Skills

The number of achieved final skills was analyzed using descriptive statistics (median and ranges). Each final skill was also analyzed separately for the number of times a skill was achieved. The correlation between the number of achieved skills in each skills-test was assessed with Kendall's rank correlation coefficient (Kendall's Tau).

Consistency

The consistency in the number of exoskeleton skills which were performed the same in the first two attempts (successful-successful or failure-failure) relative to the total number of performed skills was used as a reliability measure of the Intermediate-skills-test and the Final-skills-test. An overall consistency of >70% was considered reliable. Each intermediate skill and final skill was also analyzed separately for the number of times a skill was tested, performed consistent and performed successful.

Complications

To assess the occurrence of complications during an exoskeleton training program both the physical therapists and participants filled out a logbook after each training session including any complications. The reported complications such as the number of skin damages, location of skin damages, incidence of reported muscle or joint pain, number of incontinence problems, device related errors, fractures, venous-lymphatic stasis and falls during the exoskeleton training program were analyzed using descriptive statistics.

Results

Participants

Out of 12 participants 10 (83%) completed the training program. Reasons for not completing the training program were inability to learn the basic skills of the exoskeleton (stopped after seven training sessions and performed the first Intermediate-skills-test) and absence of perceived benefit (stopped after two training sessions and did not perform an Intermediate-skills-test). Eleven participants completed at least one Intermediate-skills-test, the data of these participants was used in the analysis of the hierarchy and consistency of the intermediate skills. Due to time constraints, one participant was not able to perform the skills a second time during the Final-skills-test. For this participant the set of final skills was repeated twice one week later. The data of the second Final-skills-test was only used for the consistency analysis whereas the first Final-skills-test was used for the analysis of the achieved skills after the training program. We do not expect that this had an impact on the outcome of the current study. An overview of the patient characteristics is given in Table 2.

TABLE 2

Table 2. Patient characteristics.

Achieved Intermediate Skills

The Friedman test revealed a significant difference between the number of achieved intermediate skills between the measurements [X_F²(2) = 12.36, p = 0.001]. Post-hoc analysis revealed that the achieved intermediate skills significantly increased from a median of 4 [1–7] at Intermediate-skills-test one to 10.5 [5–26] at Intermediate-skills-test three. There was no significant difference in the number of achieved skills between Intermediate-skills-test one and two and between two and three. Figure 2 shows the achieved intermediate skills per participant. Five out of ten participants showed the expected increase in number of achieved skills over the three measurements.

FIGURE 2

Figure 2. Achieved intermediate skills measured with the Intermediate-skills-test one, two and three. Each line represents a participants. Thick red line represents the median achieved intermediate skills. Dotted lines represent participants who did not perform the Final-skills-test.

Detailed post-hoc analysis revealed that five of the intermediate skills were achieved during all measurements (see completely green bars in Figure 3). Three out of five intermediate walking skills, and two advanced skills were achieved in approximately half of the tested times, these skills are highlighted with an “#”-sign in Figure 3.

FIGURE 3

Figure 3. Times performed consistent and achieved of each separate intermediate skill. Numbers in brackets represent the number of tested participants. Consistent = performed the same in the first two attempts (successful-successful or failure-failure). Achieved = at least two out of three successful attempts & = >70% performed consistent # = achieved in ~50% of the times.

Hierarchy of the Skills

In general, the rate of participants who achieved the intermediate skills decreased. In three skills, the “walk 10 m without stops,” “180° curve tot the right without stops,” and “open door toward”-skill, the number of participants who achieved the skills was smaller than skills later in the hierarchical order (Figure 4). The coefficient of reproducibility was 0.98 (number of scaling errors: 14, number of skills: 27 and number of observations: 31 (10 participants with three observations and one participant with one observations). The scaling errors occurred in nine different skills (Intermediate-skill 3, 4, 6, 7, 9, 10, 12, 16, and 22) and in eight out of eleven participants. Four scaling errors occurred in Intermediate-skills-test one, three in Intermediate-skills-test two, and seven in Intermediate-skills-test three.

FIGURE 4

Figure 4. Rates of achievement of each intermediate skill.

Achieved Final Skills

Eight participants were able to walk without assistance between the 18th and 23th training session and therefore met the criteria to perform the Final-skills-test. The median number of successfully performed final skills in these participants was 16.5 [13–20] and 17 [14–19] skills in the first and second time. In the Final-skills-test, 15 skills were at least one time achieved by all eight subjects (see Figure 5). The martial arts mat was not achieved by half of the participants.

FIGURE 5

Figure 5. The number of consistent performances of each final skill. Green bars represent inconsistent performances in which one out of two attempt was successful performed. Blue and red bars represent consistent performances. A = all performed consistent; N = <70% performed consistent.

The number of achieved final skills in the first and the second time were significantly correlated (r_τ = 0.75, p < 0.05) and not significantly different (z = −0.71, p = 0.75, effect size = −0.18). Table 3 revealed the correlation between the various skills-tests. The number of achieved skill in none of the Intermediate-skills-tests were significantly correlated with the achieved skills in any other skill-tests.

TABLE 3

Table 3. Correlation (Kendall's tau) between all test moments.

Consistency

Eleven participants performed in total 235 intermediate skills, of which 171 (73%) were performed the same in the first two attempts (successful-successful or failure-failure).

The number of participants who performed the skill, the number of times a skill was measured, the number of times a skill was performed consistent and the number of times a skill was achieved is shown for each intermediate skill in Figure 3. Eighteen skills were performed consistent in more than 70% of the times (highlighted with an “&”-sign in Figure 3). Of these skills, 10 skills were performed consistent during all Intermediate-skills-tests (see completely red bars in Figure 3).

Eight participant performed all twenty final skills twice resulting in a total of 160 final skills. They performed 130 (81%) final skills the same in both attempts. The median number of inconsistent performed skills per participant was 2.5 [0–9]. An overview of the consistent and inconsistent performances of each final skill is depicted in Figure 5. Most skills were performed consistently by seven (9 skills) or six (6 skills) participants. Two skills were performed consistently by all participants (highlighted with an “A”-sign in Figure 5), whereas three skills were performed inconsistent by three participants (180° curve to the right, slalom and martial arts mat) (highlighted with an “N”-sign in Figure 5).

Complications

Eight out of twelve participants experienced device related skin damage at the feet (n = 3), knee (n = 5), thigh (n = 3), pelvic (n = 4), and/or trunk (n = 1) area. In four participants the skin damage resulted in at least one missed training session, which was rescheduled at the end of the training period. In case of skin damage, extra padding was added to prevent reoccurrence of the complication. As a result, most skin damage occurred in the early phase of the training program. Seven participants reported muscle or joint pain during the training program around the hands/wrists (n = 2), arms (n = 3), shoulders (n = 3), neck (n = 3), trunk (n = 1), and/or back (n = 3). None of the complications evolved into serious adverse events. During the entire study the incidence of device related errors was three times in 218 training sessions. No incontinence problems, fractures, venous-lymphatic stasis or falls were mentioned by the participants or physical therapists in the study.

Discussion

In the present study, a framework for measuring the ability to perform basic and advanced exoskeleton skills throughout an exoskeleton training program was developed and tested. Ten participants completed the training program and were tested during the 2nd, 4th, and 6th training week. They showed an increase in the achieved intermediate skills from four at the first to 10.5 at the third Intermediate-skills-test. The rate of participants who achieved the intermediate skills decreased and the coefficient of reproducibility was 0.98. In the last training week, eight participants successfully performed 16.5 and 17 skills in the Final-skills-test. An overall consistency of 73% in the Intermediate-skills-test and 81% in the Final-skills-test was achieved.

Similar to other assistive technologies, such as prostheses and to a lesser extent wheelchairs, exoskeleton community use is preceded with a training program. Although the type and extent of the risks of exoskeletons are yet to be understood (He et al., 2017), the risks associated with exoskeleton use seems higher than with prostheses or wheelchair. The advantage of a prostheses and a wheelchair is that it can be used independently for ambulation in an early phase. Therefore, clinical tests such as the timed up and go-test (Condie et al., 2006) or the mechanical efficiency (Leving et al., 2016) can be used to evaluate the progress in performance in using the assistive technologies. In contrast, most people are not able to perform basic ambulation skills at the start of an exoskeleton training program and multiple training sessions are needed before independent ambulation is possible. Assessing performance in using assistive technologies can also be done with standardized skills-tests. For wheeled mobility several skills-test, such as the wheelchair skills-test, are available (Fliess-Douer et al., 2010). Until now there were no standardized skills-test for exoskeleton performance. Several studies marked the training session in which a skill with an exoskeleton was performed with varying levels of trainer assistance (Spungen et al., 2013; Hartigan et al., 2015; Kozlowski et al., 2015; Benson et al., 2016; Platz et al., 2016). However, in these studies the performed exoskeleton skills were kept up in a logbook and the independent achievement of skills was not tested on a regular bases. The main objective of this study was to develop a framework for measuring the ability to perform basic and advanced exoskeleton skills throughout an exoskeleton training program.

A framework to assess the progress of exoskeleton skills should consist of tests measuring achieved skills in a hierarchical order of difficulty. As a consequence participants should progress during an intensive exoskeleton training program. Although the Friedman test revealed a significant difference in the number of achieved skills between the Intermediate-skills-tests, these results should be interpreted with care because several tied ranks were observed and a small number of participants were included. Nevertheless, the number of achieved intermediate skills significantly increased from 4 at the first to 10.5 at the third Intermediate-skills-test. Furthermore, nine and seven out of ten participants had an increase in the number of achieved skills between the first and second and the second and third Intermediate-skills-test, respectively. However, such an increase in number of achieved skills doesn't automatically indicate that the skills are in a correct order of difficulty. In the current study, two measures were used to assess the hierarchy. According to the coefficient of reproducibility (0.98) the intermediate skills were in the correct order of difficulty (Guttman, 1944; Tyson and DeSouza, 2004). The rate of the number of participants achieving each skill revealed three skills (walk 10 m without stops, 180° curve to the right without stops, and open door toward-skill) that were achieved by a larger number of participants in the subsequent skill. Detailed post-hoc analysis revealed that only an unachieved “walk 10 m without stops”-skill was followed by achieved skills in more than two observations (five observations). Covering a distance of 10 m without stops was the last basic skills before advanced skills were tested. The advanced skills consisted of an additional task during walking such as a sudden stop, curves or passing a doorstep, but a shorter distance of ~3 m had to be covered. The length of the skill of walking a distance of 10 m increased the chance of errors and not achieving the skill. Two other studies (Spungen en Platz) also recorded the moment the 10 m walking skill was performed without assistance, four (Spungen et al., 2013) and one (Platz et al., 2016) out of seven participants were able to perform the skill independent within 24 training sessions. Indicating the difficulty of walking 10 m without assistance. In the current study each test session was scheduled in advance within a training session and an extra person was present during the skills-test. As a result, most participants were more stressed during the skills-test than during other training sessions. An increased stress level could evoke more spasticity (Fleuren et al., 2009) causing involuntary stops, which particularly influenced the achievement of the “10 m walking skill without stops”-skill. Because walking without stops is crucial in performing most advanced skills, we prefer to keep the proposed order of the skills. However, for future research we would advise to change the order of the Intermediate-skills for the “walking curves” according to the preference of the patient.

In addition to assess the progression, the framework had to discriminate across participants. In all Intermediate-skills-tests, differences between participants were apparent. After 2 weeks of exoskeleton training participants were only able to perform basic skills, but varied between standing and walking skills. This was in line with the findings of Spungen et al. (2013). In the current study, all participants were able to perform all intermediate standing skills without assistance after 4 and 6 weeks of training, but differed in walking and advanced skills. In addition to differences across participants at the Intermediate-skills-tests, participants showed various learning curves. The low correlation between the three Intermediate-skills-tests (r_τ between −0.15 and 0.37) supports the various learning process across participants. In conclusion, the framework proposed in this study measured the progress in the ability to perform basic and advanced exoskeleton skills, had the skills in a hierarchical order of difficulty and could discriminate across participants.

A second important prerequisite of the framework is that the tested skills were reliable. An overall consistency of 73% in the Intermediate-skills-test and 81% in the Final-skills-test was achieved. Detailed analysis revealed several skills that had a consistency of <70% (see Figures 3, 5). Remarkable was a lower consistency for the same skills in the Intermediate-skill-test compared to the Final-skill-test or vice versa. A consistency below 70% in the Intermediate-skill-test whereas a consistency above 70% in the Final-skill-test was met for the skills: sit-to-stand, stopping with the preferred or not preferred leg, arresting gait at command, passing a narrow passage, and pivot-turn to sit. The lower consistency of the intermediate skills were possibly due to the learning process. During the Intermediate-skills-test participants had to perform skills they practiced only once or twice without assistance. As a result, the skill was sometimes successfully performed by chance instead of competence and therefore participants were unable to perform the skill consistent. Therefore, we recommend that a skill should be performed at least two times when tested. The skills 180° curve to the right, slalom and martial arts mat had a low consistency in the Final-skills-test whereas a high consistency was obtained in the Intermediate-skills-test. These skills were performed by only a minority of the participants during the Intermediate-skills-tests indicating that most participants were only in the last training sessions at a level that they could practice these advanced skills. Nevertheless, the majority of the tested skills were performed consistent in the Intermediate- and Final-skills-test. Therefore, considering a skill achieved after two out of three successful attempts seems a good assumption to evaluate the skill-level.

In order to achieve exoskeleton skills, participants received multiple training sessions per week over an 8 week period. Such an intensive training program yields the potential of complications such as bruises and other skin damage. Most previous studies indicated that in hospital training with an exoskeleton was safe (Zeilig et al., 2012; Esquenazi, 2013; Kolakowsky-Hayner et al., 2013; Kubota et al., 2013; Spungen et al., 2013; Miller et al., 2016). Although other studies disclosed mild to moderate skin damages in half of the participants (five out of ten) (Benson et al., 2016) (four out of seven) (Platz et al., 2016), the intensity (session per week) and duration of the training period in this study was similar to most previous studies (Spungen et al., 2013; Benson et al., 2016; Miller et al., 2016; Platz et al., 2016). In the current study, eight out of twelve participants experienced skin damage during the training program. In four cases this skin damage resulted in at least one missed training. Whereas all skin damages reported in the study of Benson and none of the skin damages reported by Platz led to discontinuation of the training (Benson et al., 2016; Platz et al., 2016). Because of extra padding in the early phase of the training program, skin damage rarely occurred in the later phase. In addition, during the whole training program special care was taken to the correct joint alignment. Therefore, none of the patients had to reduce the training intensity in the later phase of the training program. Moreover, it suggests that special attention to joint alignment and padding during the training reduces the risk of skin damage.

In the current study, the Rewalk exoskeleton was used. In the current study, the ReWalk exoskeleton was used. Nowadays, there are multiple exoskeletons available on the market, which have their specific interaction with and control of the exoskeleton. As a consequence the hierarchical order of the skills in the framework might be slightly different between exoskeletons. The main difference between the current available exoskeletons for home and community is the control of initiation of gait, arresting of gait, and involuntary stops. For example, to initiate gait, the ReWalk and Ekso require a forward and lateral shift of the trunk, the Indego exoskeleton requires a forward trunk excursion, and the Rex exoskeleton does not require any trunk movement (Bryce et al., 2015). Despite the differences in interaction with and control of the exoskeleton, all skills proposed in the current study are relevant and applicable to the current available exoskeletons. Standing and walking skills are presumed to be achieved before users can perform additional advanced skills, which are mostly performed during walking. The first eight advanced skills, require less interaction with the environment. Within these skills a distinction was made in the fluency of the performance (with or without stops). Therefore, we expect that the hierarchical order of the first 18 intermediate skills can be applied to other exoskeletons. The hierarchical order of the last nine advanced skills might be slightly different across exoskeletons due to the difference in interaction with and control of the exoskeleton. However, achieving one of these skills indicates a highly advanced exoskeleton skill level.

The skills-tests proposed in this framework were based on independent performance of exoskeleton skills, but did not take the quality of the performance into consideration. For future research, the quality of how a skill is performed might be of interest in addition to if it is possible to perform a skill independent. Moreover, all exoskeleton skills in this study were assessed in a clinical setting and it remains unknown which skills are relevant and which skill-level is necessary for safe community use. Therefore, future research should focus on community use of an exoskeleton and its relation to the skill-level during the training period measured with the proposed framework.

Conclusion

The framework proposed in this study measured the progress in performing basic and advanced exoskeleton skills during a training program. The hierarchical ordered skills-test could discriminate across participants' skill-level. The overall consistency of the performed exoskeleton skills was considered acceptable.

Disclosure

The exoskeleton training of the physiotherapists by ReWalk Robotics was given before the start of the study and ReWalk Robotics does not have any influence on the entire study.

Author Contributions

All authors were involved in the design of the study. RvD, HR, and NK conceived and planned the experiments. RvD performed the data collection during the experiments. RvD and NK processed the experimental data, performed the analysis, drafted the manuscript and designed the figures. IvN, HvdM, and NK supervised the project. All authors discussed the results and contributed to the final manuscript.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fnins.2017.00699/full#supplementary-material

References

Adriaansen, J. J., Ruijs, L. E., van Koppenhagen, C. F., van Asbeck, F. W., Snoek, G. J., van Kuppevelt, D., et al. (2016). Secondary health conditions and quality of life in persons living with spinal cord injury for at least ten years. J. Rehabil. Med. 48, 853–860. doi: 10.2340/16501977-2166

PubMed Abstract | CrossRef Full Text | Google Scholar

Benson, I., Hart, K., Tussler, D., and van Middendorp, J. J. (2016). Lower-limb exoskeletons for individuals with chronic spinal cord injury: findings from a feasibility study. Clin. Rehabil. 30, 73–84. doi: 10.1177/0269215515575166

PubMed Abstract | CrossRef Full Text | Google Scholar

Bryce, T. N., Dijkers, M. P., and Kozlowski, A. J. (2015). Framework for assessment of the usability of lower-extremity robotic exoskeletal orthoses. Am. J. Phys. Med. Rehabil. 94, 1000–1014. doi: 10.1097/PHM.0000000000000321

PubMed Abstract | CrossRef Full Text | Google Scholar

Condie, E., Scott, H., and Treweek, S. (2006). Lower limb prosthetic outcome measures: a review of the literature 1995 to 2005. J. Prosthet. Orthotics 18, 13–45. doi: 10.1097/00008526-200601001-00004

CrossRef Full Text | Google Scholar

Dunn, R. B., Walter, J. S., Lucero, Y., Weaver, F., Langbein, E., Fehr, L., et al. (1998). Follow-up assessment of standing mobility device users. Assist. Technol. 10, 84–93. doi: 10.1080/10400435.1998.10131966

PubMed Abstract | CrossRef Full Text | Google Scholar

Esquenazi, A. (2013). New bipedal locomotion option for individuals with thoracic level motor complete spinal cord injury. J. Spinal Res. Fundation 8, 26–28. Available online at: http://www.ryzur.com.cn/uploadfile/2016/0830/20160830115943895.pdf

Federici, S., Meloni, F., Bracalenti, M., and De Filippis, M. L. (2015). The effectiveness of powered, active lower limb exoskeletons in neurorehabilitation: a systematic review. Neurorehabilitation 37, 321–340. doi: 10.3233/NRE-151265

PubMed Abstract | CrossRef Full Text | Google Scholar

Fleuren, J. F., Voerman, G. E., Snoek, G. J., Nene, A. V., Rietman, J. S., and Hermens, H. J. (2009). Perception of lower limb spasticity in patients with spinal cord injury. Spinal Cord 47, 396–400. doi: 10.1038/sc.2008.153

PubMed Abstract | CrossRef Full Text | Google Scholar

Fliess-Douer, O., Vandelandewijck, Y. C., Lubel Manor, G., and Van Der Woude, L. H. (2010). A systematic review of wheelchair skills tests for manual wheelchair users with a spinal cord injury : towards a standardized outcome measure. Clin. Rehabil. 24, 867–886. doi: 10.1177/0269215510367981

PubMed Abstract | CrossRef Full Text | Google Scholar

Guttman, L. (1944). A basis for scaling qualitative data. Am. Sociol. Rev. 9, 139–150. doi: 10.2307/2086306

CrossRef Full Text | Google Scholar

Hartigan, C., Kandilakis, C., Dalley, S., Clausen, M., Wilson, E., Morrison, S., et al. (2015). Mobility outcomes following five training sessions with a powered exoskeleton. Top. Spinal Cord InJ. Rehabil. 21, 93–99. doi: 10.1310/sci2102-93

PubMed Abstract | CrossRef Full Text | Google Scholar

He, Y., Eguren, D., Luu, T. P., and Contreras-Vidal, J. L. (2017). Risk management and regulations for lower limb medical exoskeletons : a review. Med. Devices 10, 89–107. doi: 10.2147/MDER.S107134

PubMed Abstract | CrossRef Full Text | Google Scholar

Jensen, M. P., Truitt, A. R., Schomer, K. G., Yorkston, K. M., Baylor, C., and Molton, I. R. (2013). Frequency and age effects of secondary health conditions in individuals with spinal cord injury: a scoping review. Spinal Cord 51, 882–892. doi: 10.1038/sc.2013.112

PubMed Abstract | CrossRef Full Text | Google Scholar

Kolakowsky-Hayner, S. A., Crew, J., Moran, S., and Shah, A. (2013). Safety and feasibility of using the EksoTM bionic exoskeleton to aid ambulation after spinal cord injury. J. Spine 4, 1–8. doi: 10.4172/2165-7939.S4-003

CrossRef Full Text | Google Scholar

Kozlowski, A. J., Bryce, T. N., and Dijkers, M. P. (2015). Time and effort required by persons with spinal cord injury to learn to use a powered exoskeleton for assisted walking. Top. Spinal Cord Inj. Rehabil. 21, 110–121. doi: 10.1310/sci2102-110

PubMed Abstract | CrossRef Full Text | Google Scholar

Kubota, S., Nakata, Y., Eguchi, K., Kawamoto, H., Kamibayashi, K., Sakane, M., et al. (2013). Feasibility of rehabilitation training with a newly developed wearable robot for patients with limited mobility. Arch. Phys. Med. Rehabil. 94, 1080–1087. doi: 10.1016/j.apmr.2012.12.020

PubMed Abstract | CrossRef Full Text | Google Scholar

Lee, B. B., Cripps, R. A., Fitzharris, M., and Wing, P. C. (2014). The global map for traumatic spinal cord injury epidemiology: update 2011, global incidence rate. Spinal Cord 52, 110–116. doi: 10.1038/sc.2012.158

PubMed Abstract | CrossRef Full Text | Google Scholar

Leving, M. T., Vegter, R. J., de Groot, S., and van der Woude, L. H. (2016). Effects of variable practice on the motor learning outcomes in manual wheelchair propulsion. J. Neuroeng. Rehabil. 13, 1–15. doi: 10.1186/s12984-016-0209-7

PubMed Abstract | CrossRef Full Text | Google Scholar

Louie, D. R., Eng, J. J., and Lam, T. and Spinal Cord Injury Research Team (SCIRE). (2015). Gait speed using powered robotic exoskeletons after spinal cord injury: a systematic review and correlational study. J. Neuroeng, Rehabil. 12, 1–10. doi: 10.1186/s12984-015-0074-9

PubMed Abstract | CrossRef Full Text | Google Scholar

Middleton, J. W., Yeo, J. D., Blanch, L., Vare, V., Peterson, K., and Brigden, K. (1997). Clinical evaluation of a new orthosis, the “Walkabout”, for restoration of functional standing and short distance mobility in spinal paralysed individuals. Spinal Cord 35, 574–579. doi: 10.1038/sj.sc.3100459

PubMed Abstract | CrossRef Full Text | Google Scholar

Miller, L. E., Zimmermann, A. K., and Herbert, W. G. (2016). Clinical effectiveness and safety of powered exoskeleton-assisted walking in patients with spinal cord injury : systematic review with meta- analysis. Med. Devices 9, 455–466. doi: 10.2147/MDER.S103102

PubMed Abstract | CrossRef Full Text | Google Scholar

Mirbagheri, M. M., Kindig, M. W., and Niu, X. (2015). Effects of robotic-locomotor training on stretch reflex function and muscular properties in individuals with spinal cord injury. Clin. Neurophysiol. 126, 997–1006. doi: 10.1016/j.clinph.2014.09.010

PubMed Abstract | CrossRef Full Text | Google Scholar

Onose, G., Cârdei, V., Craciunoiu, S., Avramescu, V., Opris, I., Lebedev, M. A., et al. (2016). Mechatronic wearable exoskeletons for bionic bipedal standing and walking : a new synthetic approach. Front. Neurosci. 10:343. doi: 10.3389/fnins.2016.00343

PubMed Abstract | CrossRef Full Text | Google Scholar

Platz, T., Gillner, A., Borgwaldt, N., Kroll, S., and Roschka, S. (2016). Device-training for individuals with thoracic and lumbar spinal cord injury using a powered exoskeleton for technically assisted mobility : achievements and user satisfaction. Biomed. Res. Int. 2016, 1–10. doi: 10.1155/2016/8459018

PubMed Abstract | CrossRef Full Text | Google Scholar

Spungen, A. M., Asselin, P. K., Fineberg, D. B., Kornfeld, S. D., and Harel, N. Y. (2013). “Exoskeletal-assisted walking for persons with motor-complete paraplegia,” in Force Sustainment: Rehabilitation, Regeneration and Prosthetics for Re-Integration to Duty, 6–1; 6–14. STO-MP-HFM-228. Available online at: http://www.ryzur.com.cn/uploadfile/2016/0830/20160830115519272.pdf

PubMed Abstract | Google Scholar

Tyson, S. F., and DeSouza, L. H. (2004). Development of the brunel balance assessment: a new measure of balance disability post stroke. Clin. Rehabil. 18, 801–810. doi: 10.1191/0269215504cr744oa

PubMed Abstract | CrossRef Full Text | Google Scholar

Wyndaele, M., and Wyndaele, J. J. (2006). Incidence, prevalence and epidemiology of spinal cord injury: what learns a worldwide literature survey? Spinal Cord 44, 523–529. doi: 10.1038/sj.sc.3101893

PubMed Abstract | CrossRef Full Text | Google Scholar

Yang, A., Asselin, P., Knezevic, S., Kornfeld, S., and Spungen, A. M. (2015). Assessment of in-hospital walking velocity and level of assistance in a powered exoskeleton in persons with spinal cord injury. Top. Spinal Cord Inj. Rehabil. 21, 100–109. doi: 10.1310/sci2102-100

PubMed Abstract | CrossRef Full Text | Google Scholar

Zeilig, G., Weingarden, H., Zwecker, M., Dudkiewicz, I., Bloch, A., and Esquenazi, A. (2012). Safety and tolerance of the ReWalkTM exoskeleton suit for ambulation by people with complete spinal cord injury: a pilot study. J. Spinal Cord Med. 35, 96–101. doi: 10.1179/2045772312Y.0000000003

PubMed Abstract | CrossRef Full Text | Google Scholar