The Eurobench platform is the central site where all benchmarking material is brought to the community. We can highlight two key pillars: the access to protocols, and the benchmark of experiments following these protocols.
The different protocols gathered in Eurobench are available from the platform, which is a front-end to a database containing and linking to appropriate documentation. A large majority of the protocols developed in Eurobench are uploaded and accessible from the website.
The experiment benchmarking relies on a set of benchmarking algorithms that have been developed by around 20 different entities. Targeting harmonization or standardization of the practices and software methods, we designed a guide presenting the benchmarking principles of Eurobench, and collected experimental data format, promoting as much as possible the use of these principles in the design of the algorithms. Most of the metrics associated to the protocols are directly accessible from the Eurobench GitHub space and can be used by any interested person, with a spirit of transparency and open collaboration, respecting the developer licenses.
Figure 1: Documentation of the Eurobench software methodology
Figure 2: The Eurobench space on GitHub, a collection of software tools developed in Eurobench, including several protocol metrics
Based on this harmonization, we developed appropriate tools for enabling an automatic benchmark of experimental data, process common for all Eurobench protocols. This automation mechanism is deployed onto the Eurobench platform.
Figure 3: Illustration of a protocol as presented on the Eurobench Benchmarking Platform
Projects involved
There are 14 projects and more than 20 entities involved in developing and implementing the Eurobench platform. The following projects are listed below:
– DYSTURBANCE: Dynamic and static pusher to benchmark balance. A stability analysis under external perturbation.
– Bench: a Biomechanical tEsting platform for of sit to staNd assessment with an instrumented Chair.
– STEPbySTEP: Systematic Test of Exoskeleton Products by a Stairs-based Testbed Evaluation Protocol
– UDBenchmark: Ups and Downs: benchmarking gait stability of wearable robotic assistive devices and humanoid robots on slopes and lateral inclines
– PEPATO: PErformance indicators of spatiotemporal PATterns of the spinal muscle coordination Output during walking with an exoskeleton
– BeStaBle: Benchmarking System for Assessment of Balance Performance
– COMTEST: Complete Modular Test Stand for Human and Humanoid Posture Control and Balance
– TestEd: Benchmark for Benchmarking Industrial Exoskeleton Technology: Testbed and Software routine development for scenario “Moving in Narrow Space”
– B.E.A.T.: Balance Evaluation Automated Testbed
– MADROB: Modular Active Door for RObot Benchmarking
– BEAST: Benchmark enabling active shopping trolley beast
– BULLET: Benchmarking Upper Limbs Loads on Exoskeleton Testbeds
– BenchBalance: A device to apply well defined perturbations for benchmarking balance capabilities of wearable robots