Vectorized rendering

[jstmn]

Howdy,

Is there someway to do vectorized rendering? Specifically, I would like to get 3 images from all N robots/environments. The only visualization functionality i've seen is MujocoVisualizer.

Otherwise is there support for madrona or something similar? https://github.com/shacklettbp/madrona

Thanks

[flferretti]

Hi @jstmn, thanks for reaching out.

If you want to visualize the same model under different poses, you can take a look at the second-to-last cell in this example notebook: https://github.com/gbionics/jaxsim/blob/main/examples/jaxsim_as_physics_engine_advanced.ipynb.

At the moment, rendering isn't vectorized, so workflows like generating images from N environments need to be handled manually (e.g., looping over states). That said, I'm planning to introduce a small helper utility to make less tedious soon.

Regarding Madrona, there's no integration at the moment.

If you can share a more focused example of what you're trying to render, I can help sketch a solution

[jstmn]

Hey, thanks for the info. My use case is to do RGB based model predictive control, so I need to have N environments each with M cameras take H simulation steps, and then get an image from every environment x camera x rollout step (N * M * H total images). This happens at every planning step. Because of these rendering requirements it doesn't seem like this would be the right library for my use case! Maybe sequential rendering is really fast though.

[flferretti]

That makes sense, thanks for the explanation. Keep in mind is that JaxSim is a physics engine and MuJoCo is only used for visualization. This means the rendering pipeline isn't designed to be GPU-efficient anyway, as it would involve host–device transfers.

Given your workload, rendering would currently be sequential and likely become a bottleneck. The simulation side is fully vectorized, so you could still decouple things (e.g., async rendering only a subset of trajectories or at a lower frequency). But if you truly need all frames at every step, then I agree it's probably not the right tool for that use case in its current form.

If you're curious, it might still be worth benchmarking sequential rendering with your target numbers, but I wouldn’t expect it to scale well to that regime