UnifoLM-VLA deployment in ISAACSIM not working

[EdoCavaReply]

Hi Unitree team.
We are trying to deploy the Unifolm-VLA model in IsaacSim (NOT ISAACLAB) using a custom simulation built from scratch. We use a G1 29dof robot with dex1 in an environment which closely resembles the one used in the training dataset.

The proprio states we collect (alongside the 4 video streams high_left high_right wrist_left and wrist_right) are 23 (as indicated in configs.py): left_EE_XYZ, left_EE_r6, left_gripper, right_EE_XYZ, right_EE_r6, right_gripper, waist_RPY. Does the models returns the same structure?

We performed three tests to evaluate the model in simulation (relate to the corresponding videos):

1. we use the observations from the stack block datasets episode 0: the 29 body observations are applied to the g1 joints in order to assert the data simulation correctness (without using the VLA at all but only for testing purposes)
2. we feed the robot with the images from episode 0 (extracting the frames from the videos) while providing the actual proprio observations of the robot from simulation
3. we feed the robot the camera streams from the simulation along with the proprio from the simulation

as you can see from video1 the robot replicates exactly the same movements in the dataset: this proves the simulation is well constructed
but as you can see in video 2 and 3 the robot fails to perform the task and is looping in a little movement.

G1_replicating_obs.mp4

G1_images_from_dataset.mp4

G1_cameras_ord1.mp4

Where are we mistaking?
What cloud be the possible cause of this behavior?
What can we adjust?
Have you any solutions for us?

[Frfor17]

its kinda issue with a camera observation streaming or something, maybe sim not getting the actual camera frames

[Frfor17]

Last 2 videos seems to have limited max_steps?