Towards Using Multiple Iterated, Reproduced, and Replicated Experiments with Robots (MIRRER) for Evaluation and Benchmarking

2024/08/08 by Adam Norton, Norton, Adam, Brian W. Flynn +1 · 1 citation
Engineering · Computer Science · #Robot Manipulation and Learning #Reinforcement Learning in Robotics #Modular Robots and Swarm Intelligence

paper · pdf · doi:10.48550/arxiv.2408.04736

Abstract

The robotics research field lacks formalized definitions and frameworks for evaluating advanced capabilities including generalizability (the ability for robots to perform tasks under varied contexts) and reproducibility (the performance of a reproduced robot capability in different labs under the same experimental conditions). This paper presents an initial conceptual framework, MIRRER, that unites the concepts of performance evaluation, benchmarking, and reproduced/replicated experimentation in order to facilitate comparable robotics research. Several open issues with the application of the framework are also presented.

Cited by

• Physical AI Governance: From Theory to Practice Across Life Cycle

Related

• BenchBot: Evaluating Robotics Research in Photorealistic 3D Simulation and on Real Robots
• Integrated Benchmarking and Design for Reproducible and Accessible Evaluation of Robotic Agents
• RB2: Robotic Manipulation Benchmarking with a Twist
• A Survey on Reproducibility by Evaluating Deep Reinforcement Learning\n Algorithms on Real-World Robots
• Robot Learning as an Empirical Science: Best Practices for Policy Evaluation