RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins (early version)

Yao Mu ^{* 1 3}, Tianxing Chen ^{* 1 3 4}, Shijia Peng ^{* 2 4}, Zanxin Chen ^{* 2 4},
Zeyu Gao ⁵, Yude Zou ⁴, Lunkai Lin ², Zhiqiang Xie ², Ping Luo^{† 1}

¹ The University of Hong Kong,
² AgileX Robotics,
³ Shanghai AI Laboratory,

⁴ Shenzhen University,
⁵ Institute of Automation, Chinese Academy of Sciences

^*Equal Contribution, ^†Corresponding authors

ECCV Workshop 2024 (Best Paper)

arXiv

Code

Dataset

The RoboTwin leaderboard is powered by the D-Robotics cloud system.

Abstract

Effective collaboration of dual-arm robots and their tool use capabilities are increasingly important areas in the advancement of robotics. These skills play a significant role in expanding robots' ability to operate in diverse real-world environments. However, progress is impeded by the scarcity of specialized training data. This paper introduces RoboTwin, a novel benchmark dataset combining real-world teleoperated data with synthetic data from digital twins, designed for dual-arm robotic scenarios. Using the COBOT Magic platform, we have collected diverse data on tool usage and human-robot interaction. We present a innovative approach to creating digital twins using AI-generated content, transforming 2D images into detailed 3D models. Furthermore, we utilize large language models to generate expert-level training data and task-specific pose sequences oriented toward functionality. Our key contributions are: 1) the RoboTwin benchmark dataset, 2) an efficient real-to-simulation pipeline, and 3) the use of language models for automatic expert-level data generation. These advancements are designed to address the shortage of robotic training data, potentially accelerating the development of more capable and versatile robotic systems for a wide range of real-world applications.

RoboTwin's AIGC & Expert Data Generation pipeline

RoboTwin's AIGC & Expert Data Generation pipeline RoboTwin's pipeline for AIGC and expert data generation involves automatically extracting object segmentation and textual descriptions from a single RGB photo. This is followed by generating 3D geometry, surface normals, wireframes, and texture maps to create a high-fidelity simulation object. With the object's surface normal and pose information, we can decompose and generate grasping postures. Furthermore, we utilize the capabilities of large models to generate expert data for tasks in a zero-shot manner.

(d) right

Digital Twin

We assign specific coordinate axes to the functional parts of objects within the model. For instance, for a hammer, one axis is aligned with the hammerhead, identifying the functional part, while another axis indicates the approach direction. This strategic alignment is crucial for automating the calculation of grasp poses, ensuring essential robotic manipulation and tool usage. Grasp poses are computed perpendicular to the surface normal of the functional part along the designated approach direction axis, ensuring correct and efficient tool use with minimal manual intervention.

Example:

Ping Pong Sweep (Real)

(a) left

(b) top

Block Sweep (Sim)

(a) left

(b) top

Empty Cup Place (Real)

(a) left

(b) top

Empty Cup Place (Sim)

(a) left

(b) top

Dataset [Video Show]

Hardware Platform

COBOT Magic platform from AgileX Robotics (https://global.agilex.ai/products/cobot-magic) For the acquisition of real-world data, we employed the open-source COBOT Magic platform from AgileX Robotics, which is equipped with four AgileX Arms and four Intel Realsense D-435 RGBD cameras and is built on the Tracer chassis. These cameras are strategically positioned: one on the high part of the stand for an expansive field of view, two on the wrists of the robot's arms, and one on the low part of the stand which is optional for use. The front, left, and right cameras capture data simultaneously at a frequency of 30Hz. The data collection and alignment are facilitated by tools provided by the ARIO Data Alliance, available at our GitHub repository. Each captured frame consists of three images from the cameras, each providing an RGB and depth image at a resolution of 640 x 480 pixels. Additionally, the data includes the poses of the robotic arms' joints and end-effectors for both master and slave configurations, encompassing both left and right arms. For motion tasks, the linear and angular velocities of the differential drive chassis are also recorded. All data storage and formatting adhere to the unified standards established by the ARIO Data Alliance.

Experiment

Experimental Result Our experimental aim is not to delve into the design choices of different strategy networks, but to explore the correctness and effectiveness of our Benchmark expert data. Our experiments are intended to verify: a) the rationality of the COBOT Magic platform settings, b) the effectiveness of the automatically generated expert data.

Bibtex

      
@article{mu2024robotwin,
  title={RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins (early version)},
  author={Mu, Yao and Chen, Tianxing and Peng, Shijia and Chen, Zanxin and Gao, Zeyu and Zou, Yude and Lin, Lunkai and Xie, Zhiqiang and Luo, Ping},
  journal={arXiv preprint arXiv:2409.02920},
  year={2024}
}

Video Show of DP3 deployment

Hammer Beat

(a) observer success

(b) top success

(d) top fail

Block Handover

(a) observer success

(b) top success

(d) top fail

Apple Cabinet Storage

(a) observer success

(b) top success

(d) top fail

Dual-Bottles Pick

(a) observer success

(b) top success

(d) top fail

Empty Cup Place

(a) observer success

(b) top success

(d) top fail

Block Sweep

(a) observer success

(b) top success

(d) top fail

Acknowledgements

The authors extend their profound gratitude to D-robotics for their invaluable support in supplying the necessary cloud computing resources that facilitated the execution of this research. Furthermore, our sincere appreciation is extended to Deeoms for their contribution in providing essential model support, which was pivotal to the successful completion of this study.