Energy-efficient and damage-recovery slithering gait design for a snake-like robot based on reinforcement learning and inverse reinforcement learning
Copyright © 2020 Elsevier Ltd. All rights reserved..
Similar to real snakes in nature, the flexible trunks of snake-like robots enhance their movement capabilities and adaptabilities in diverse environments. However, this flexibility corresponds to a complex control task involving highly redundant degrees of freedom, where traditional model-based methods usually fail to propel the robots energy-efficiently and adaptively to unforeseeable joint damage. In this work, we present an approach for designing an energy-efficient and damage-recovery slithering gait for a snake-like robot using the reinforcement learning (RL) algorithm and the inverse reinforcement learning (IRL) algorithm. Specifically, we first present an RL-based controller for generating locomotion gaits at a wide range of velocities, which is trained using the proximal policy optimization (PPO) algorithm. Then, by taking the RL-based controller as an expert and collecting trajectories from it, we train an IRL-based controller using the adversarial inverse reinforcement learning (AIRL) algorithm. For the purpose of comparison, a traditional parameterized gait controller is presented as the baseline and the parameter sets are optimized using the grid search and Bayesian optimization algorithm. Based on the analysis of the simulation results, we first demonstrate that this RL-based controller exhibits very natural and adaptive movements, which are also substantially more energy-efficient than the gaits generated by the parameterized controller. We then demonstrate that the IRL-based controller cannot only exhibit similar performances as the RL-based controller, but can also recover from the unpredictable damage body joints and still outperform the model-based controller, which has an undamaged body, in terms of energy efficiency. Videos can be viewed at https://videoviewsite.wixsite.com/rlsnake.
| Medienart: |
E-Artikel |
|---|
| Erscheinungsjahr: |
2020 |
|---|---|
| Erschienen: |
2020 |
| Enthalten in: |
Zur Gesamtaufnahme - volume:129 |
|---|---|
| Enthalten in: |
Neural networks : the official journal of the International Neural Network Society - 129(2020) vom: 02. Sept., Seite 323-333 |
| Sprache: |
Englisch |
|---|
| Beteiligte Personen: |
Bing, Zhenshan [VerfasserIn] |
|---|
| Links: |
|---|
| Themen: |
Damage recovery |
|---|
| Anmerkungen: |
Date Completed 16.11.2020 Date Revised 16.11.2020 published: Print-Electronic Citation Status MEDLINE |
|---|
| doi: |
10.1016/j.neunet.2020.05.029 |
|---|
| funding: |
|
|---|---|
| Förderinstitution / Projekttitel: |
|
| PPN (Katalog-ID): |
NLM311720730 |
|---|
| LEADER | 01000naa a22002652 4500 | ||
|---|---|---|---|
| 001 | NLM311720730 | ||
| 003 | DE-627 | ||
| 005 | 20231225142850.0 | ||
| 007 | cr uuu---uuuuu | ||
| 008 | 231225s2020 xx |||||o 00| ||eng c | ||
| 024 | 7 | |a 10.1016/j.neunet.2020.05.029 |2 doi | |
| 028 | 5 | 2 | |a pubmed24n1039.xml |
| 035 | |a (DE-627)NLM311720730 | ||
| 035 | |a (NLM)32593929 | ||
| 035 | |a (PII)S0893-6080(20)30199-4 | ||
| 040 | |a DE-627 |b ger |c DE-627 |e rakwb | ||
| 041 | |a eng | ||
| 100 | 1 | |a Bing, Zhenshan |e verfasserin |4 aut | |
| 245 | 1 | 0 | |a Energy-efficient and damage-recovery slithering gait design for a snake-like robot based on reinforcement learning and inverse reinforcement learning |
| 264 | 1 | |c 2020 | |
| 336 | |a Text |b txt |2 rdacontent | ||
| 337 | |a ƒaComputermedien |b c |2 rdamedia | ||
| 338 | |a ƒa Online-Ressource |b cr |2 rdacarrier | ||
| 500 | |a Date Completed 16.11.2020 | ||
| 500 | |a Date Revised 16.11.2020 | ||
| 500 | |a published: Print-Electronic | ||
| 500 | |a Citation Status MEDLINE | ||
| 520 | |a Copyright © 2020 Elsevier Ltd. All rights reserved. | ||
| 520 | |a Similar to real snakes in nature, the flexible trunks of snake-like robots enhance their movement capabilities and adaptabilities in diverse environments. However, this flexibility corresponds to a complex control task involving highly redundant degrees of freedom, where traditional model-based methods usually fail to propel the robots energy-efficiently and adaptively to unforeseeable joint damage. In this work, we present an approach for designing an energy-efficient and damage-recovery slithering gait for a snake-like robot using the reinforcement learning (RL) algorithm and the inverse reinforcement learning (IRL) algorithm. Specifically, we first present an RL-based controller for generating locomotion gaits at a wide range of velocities, which is trained using the proximal policy optimization (PPO) algorithm. Then, by taking the RL-based controller as an expert and collecting trajectories from it, we train an IRL-based controller using the adversarial inverse reinforcement learning (AIRL) algorithm. For the purpose of comparison, a traditional parameterized gait controller is presented as the baseline and the parameter sets are optimized using the grid search and Bayesian optimization algorithm. Based on the analysis of the simulation results, we first demonstrate that this RL-based controller exhibits very natural and adaptive movements, which are also substantially more energy-efficient than the gaits generated by the parameterized controller. We then demonstrate that the IRL-based controller cannot only exhibit similar performances as the RL-based controller, but can also recover from the unpredictable damage body joints and still outperform the model-based controller, which has an undamaged body, in terms of energy efficiency. Videos can be viewed at https://videoviewsite.wixsite.com/rlsnake | ||
| 650 | 4 | |a Journal Article | |
| 650 | 4 | |a Damage recovery | |
| 650 | 4 | |a Inverse reinforcement learning | |
| 650 | 4 | |a Motion planning | |
| 650 | 4 | |a Reinforcement learning | |
| 650 | 4 | |a Snake-like robot | |
| 700 | 1 | |a Lemke, Christian |e verfasserin |4 aut | |
| 700 | 1 | |a Cheng, Long |e verfasserin |4 aut | |
| 700 | 1 | |a Huang, Kai |e verfasserin |4 aut | |
| 700 | 1 | |a Knoll, Alois |e verfasserin |4 aut | |
| 773 | 0 | 8 | |i Enthalten in |t Neural networks : the official journal of the International Neural Network Society |d 1996 |g 129(2020) vom: 02. Sept., Seite 323-333 |w (DE-627)NLM087746824 |x 1879-2782 |7 nnns |
| 773 | 1 | 8 | |g volume:129 |g year:2020 |g day:02 |g month:09 |g pages:323-333 |
| 856 | 4 | 0 | |u http://dx.doi.org/10.1016/j.neunet.2020.05.029 |3 Volltext |
| 912 | |a GBV_USEFLAG_A | ||
| 912 | |a GBV_NLM | ||
| 951 | |a AR | ||
| 952 | |d 129 |j 2020 |b 02 |c 09 |h 323-333 | ||