Motivation

Human-centred robots such as prostheses, exoskeletons and humanoids, are becoming increasingly relevant worldwide. Many prototypes are moving out of the lab into everyday applications, in a wide range of market domains. Several roadblocks exist in this process. Some of these are technical, while others are related to the lack of reliable performance/safety indicators for these devices to meet international certifications and standardization requirements. The Robotic Research Agenda[1] has emphasized benchmarking as an important instrument to assess the Technology Readiness Level (TRL) and to quantify how robotic solutions match user needs. Recent international efforts (e.g. RockIn, Robocup, Cybathlon, DARPA Robotic Challenge) have confirmed the interest of the scientific and industrial communities in evaluating (and comparing) the performance of robotic systems in real-life environments.

However, a consolidated benchmarking methodology for Robotics has not been reached yet. In this direction, the Multi Annual Roadmap for Robotics in Europe (MAR)[2] has proposed a comprehensive list of System Abilities to help quantifying the performance of a system. Nevertheless, due to the high variability of robotic applications and technologies, it is still not clear how System Abilities can be quantified and measured on a realistic and application-specific basis. In particular, we identified three main problems:

  1. In bipedal robotics, current benchmarking approaches are focused on generic, goal-level performance indicators. They are usually based on competitions, which can hardly describe performance in sufficient detail to allow recognizing and quantifying the technical causes of performance. In contrast, System Abilities have the potential to decompose the global performance into a number of measurable and meaningful components. A detailed definition of the experimental tasks and scenarios in which these abilities should be measured is still not specified (e.g. walking on rough terrain, slopes, unstable surfaces, transitions, etc.).
  2. In Robotics, the reproducibility of results from scientific publications is very difficult. The experimental protocols and methods are not sufficiently specified to be replicated by others. When existing, metrics are numerous and overlapping, and experimental procedures are defined from each lab independently and chosen according to lab-specific functionalities. This situation undermines the basis of scientific (and industrial) progress, i.e. the evaluation of the state of the art, and building on other’s research works. This situation is aggravated by the lack of standard terminology across domains and research areas.
  3. Current measurement systems and procedures used to evaluate the robotic performance are heterogeneous and not sufficiently detailed. At the same time, most of databases of robotic performance, when existing, are not standardized, i.e. they lack of task labelling, data segmentation, data format, physical objects dimensions, type of measurement system used, HW specification. All these factors prevent benchmarking tools to be widely used by researchers and professionals, undermining their acceptance in both academia and industry.

In order to fill these gaps, EUROBENCH aims to create the first European framework for the application of benchmarking methodology on robotic systems. The framework, specifically focused on bipedal robotic technologies, will include methods and tools to measure System Ability Levels on a rigorous, quantitative and replicable way. The successful achievement of this goal will put Europe at the forefront of the evaluation of robotic systems, facilitating the process of bringing innovative robotic technologies forward to market.

 

Background

The idea, structure, and part of the methods proposed in EUROBENCH are the result of the previous achievements obtained in four years of collaborative work between the following five FP7 European projects, which include a solid methodology and several test benches already validated in relevant environment.

  • H2R, Integrative approach for the emergence of bipedal locomotion, grant n. 600698.
  • BALANCE, Balance Augmentation in Locomotion, through Anticipative, Natural and Cooperative control of Exoskeletons. grant n. 601003.
  • Koroibot, Improving humanoid walking capabilities by human-inspired mathematical models, optimization and learning, grant n. 611909.
  • Walk-Man, Whole-body Adaptive Locomotion and Manipulation, grant n. 611832.
  • Biomot, Smart Wearable Robots with Bioinspired Sensory-Motor Skills, grant agreement No. 611695.

The collaboration between these projects demonstrated that wearable robots and humanoid robots share similar goals in terms of System Abilities, e.g. achieving stable, efficient and human-like locomotion[3]. The resulting benchmarking scheme received special attention from EU representatives[4] and was recently recognized by the MAR[5] as one of the most promising efforts on standardization and benchmarking in Europe.

 

For more information and the previous benchmarking scheme, please visit www.benchmarkinglocomotion.org

[1] Strategic Research Agenda (SRA) 2014-2020, “Technology assessment”, page 962-93

[2] Multi-Annual Roadmap (MAR) For Robotics in Europe Horizon 2020, Release B 02/12/2016, page 4.

[3] D. Torricelli et al.,“Benchmarking Bipedal Locomotion: A Unified Scheme for Humanoids, Wearable Robots, and Humans,” IEEE Robot. Autom. Mag., vol. 22, no. 3, 2015.

[4] Anne Bajart. “European Commission supports benchmarking”

[5] Multi-Annual Roadmap (MAR) For Robotics in Europe Horizon 2020, Release B 02/12/2016, page 166