Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World

Inside the paper

The problem

A vision model trained only on simulated images may fail on a real robot because the rendered scenes differ from camera images. Collecting and labeling enough real images can be costly.

The idea

Train an object detector on a wide variety of simulated scenes so that it learns to locate objects despite changes in appearance. The study tests whether that detector can transfer directly to physical scenes.

How it works

• Generate simulated tabletop scenes with target objects and distractors, varying their positions and textures along with lighting and camera settings.
• Render labeled images and train a neural network to predict object positions from a single camera image.
• Keep the simulated table height fixed, allowing the setup to infer positions without estimating a general unconstrained pose from one image.
• Use predicted positions with a separate motion-planning system to guide grasps on a real robot.

What it showed

The detectors achieved roughly 1.5 cm average localization error on real images of the geometric objects, with larger errors under occlusion. In grasping trials, the robot picked up the two tested geometric targets in 38 of 40 attempts; the paper also reports ablations of training variation.

Read it for

Read Section III for the simulated data and detector, then Table II and Section IV for transfer and grasping results. The related-work survey in Section II can be skimmed on a first pass.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.