Question: What prevents open-loop trajectory replay from reaching 100% success?
Open-loop replay executes the recorded commands without feedback, so a small geometric error in the reconstruction can change the outcome. Below is one such failure.
Both videos execute the same recorded robot commands and play in sync at real speed. The slider under them moves both together.
Example: Marker-to-cup
Task: put the marker into the cup. Watch the marker as it is lowered toward the cup. In the real episode, the marker goes into the cup; in the SceneRig replay, the same commands leave it beside the cup. Why: the image-to-3D model generates the marker with a slightly different height-to-width ratio than the real one. SceneRig rescales each object uniformly, which matches its overall size but cannot correct its proportions, so the gripper contacts the marker at a slightly different position than in the real episode. Because the marker is small and placing it into the cup requires high accuracy, this small difference is enough to make the replay fail.