Gaussian Collision

[CuSO4H2O5]

Hi everyone. I’m exploring the idea of leveraging 3D Gaussian Splatting (3DGS) or NeRF-based distributions not only for rendering but also as inputs for collision detection or IPC-based physics solvers.
Do you think there is a viable approach to integrate such distributions directly into a physics engine for detecting collisions or computing constraints, or is a hybrid approach (distribution + conventional solver) still the practical choice?

[stephane-caron]

On collision detection over Gaussian splats, you could for instance take a look at the formulation proposed in FOCI: Trajectory Optimization on Gaussian Splats.

It is also possible to extract meshes from 3D/2DGS scenes and use existing mesh-based approaches. There are for instance mesh-conversion utilities distributed with 2DGS. Perhaps that's what you are referring to as the "hybrid" approach?

[momo-van]

Having Gaussians for rendering/sensor and physics/collision is in our radar. Watch for Newton announcements in the next couple of weeks for more details.

@mmacklin @gdaviet for viz.