In January 2020 I started with my graduation project for the master Systems & Control. The assignment being to develop a new safe fall controller for a lower limb exoskeleton using an optimal control technique. It was free to choose the optimal control technique, so, mostly based on personal interest I decided to choose reinforcement learning as the optimal control technique.
The tackled problem was related to future use of the lower limb exoskeleton, as currently most exoskeletons are used with some safety mechanism in place (e.g. crutches). This means that the safe fall controller is mostly responsible for being as predictable and easy to catch as possible. This, however, means that these safe fall controllers heavily depend on the safety mechanisms to actually catch the human-exoskeleton system and without these in place a hard impact is to be expected. However, in the future it is quite likely that the exoskeletons are mostly used without any external safety mechanisms in place and the safe fall controller is responsible for minimising damage if the exoskeleton is unable to balance itself. The creation of this safe fall controller has been the goal of this assignment.
As mentioned earlier, this safe fall controller has been created using reinforcement learning. For this a simulation environment has been created to train a Soft Actor-Critic reinforcement learning algorithm on. The reward signal the algorithm trained upon was based upon four measures for a good fall strategy, which were found by a literature research into human and humanoid falling. Using this simulation two controllers were developed, a forward and a backward fall controller.
These fall controllers were then evaluated using a set of test cases in which the initial positions were set to various edge cases. First, qualitatively by visual inspection using a set of evaluation criteria and second, quantitatively by comparing the impact forces and velocities of the various body segments with a “locking” fall controller (which is currently used in some lower limb exoskeletons). These evaluations showed that the created controllers significantly lowered the head impact both in amount of times as well as impact force and velocity if the head is hit. The backward fall even completely negated head impact during all test cases.
During this graduation project I learned to work more independently and to keep motivated, especially from the start of the Coronacrisis onwards as all work had to be done from home. Furthermore, I learned to use reinforcement learning and problem solving for reinforcement learning as I had never used it before this project.