Question 1: Is the SimFoundry mustard failure a coincidence?

We show two more episodes of the same task, “put the mustard into the left bin.” In both episodes, SceneRig matches the real outcome and SimFoundry does not.

Videos play at 8× speed. The slider under each row moves all three videos together; a video that ends early holds its last frame.

Episode 1 (π0.5)

Watch the left bin. The real robot and SceneRig put the mustard in the left bin. In SimFoundry, the left arm knocks the left bin over at the start; the mustard is first set on the fallen bin and finally ends up in the right bin.

Real robot: success
SceneRig (ours): success
SimFoundry: failure

Episode 2 (LingBot-VLA 2.0)

Watch the left bin. The real arms push the left bin around until it tips onto its side, and the mustard stays on the table. In SceneRig, the left arm also pushes the left bin away, and the mustard ends up beside the right bin. In SimFoundry, the robot puts the mustard into the left bin.

Real robot: failure
SceneRig (ours): failure
SimFoundry: success

Question 2: When and why does a simulator succeed while the real robot fails?

We show two such episodes with different causes: in Episode 1, SimFoundry reconstructs an object too small; in Episode 2, the real camera image differs in appearance from both simulated renderings.

Episode 1 (LingBot-VLA 2.0)

Task: put everything into the bin. Watch the brown pear next to the banana. It starts near the edge of the robot’s reach. The real gripper pushes it farther away, and the task fails; SceneRig reproduces this. Why SimFoundry succeeds: it reconstructs the pear smaller than the real one, so the gripper grasps it easily.

Real robot: failure
SceneRig (ours): failure
SimFoundry: success

Episode 2 (π0.5)

Task: put all fruits on the red plate. Watch the red apple. The real robot places the banana and the pear but leaves the apple beside the plate; both simulators place all three fruits. Why the simulators succeed (our hypothesis): in the real camera image, the red apple and the red plate look alike, so the policy may not notice that the apple is still off the plate. Both simulated renderings show slightly more contrast between them.

Real robot: failure
SceneRig (ours): success
SimFoundry: success