Clarification on non-usage of GaussianFormer voxelization module

[seamie6]

We have released optimized PyTorch and Taichi voxelization implementations, which improve speed by 1.5x and 3.5x, respectively. However, I assume this may not fully meet your needs, as it does not significantly optimize GPU memory usage, and Taichi does not support gradient computation for training.

Regrettably, we found that the CUDA implementation from GaussianFormer is not strictly correct, so our plans have changed from what we originally intended.

Originally posted by @npurson in #14

Hello,
In a previous post, you responded to me with the above. Could you expand on the statement 'the CUDA implementation from GaussianFormer is not strictly correct'. Is it that you're saying their implementation is not mathematically correct, per se or that it is not optimised for usage in a model which is self-supervised using rendering?
Thank you

[npurson]

Hi,

Regarding the comment about GaussianFormer's CUDA implementation, my analysis indicates that their Gaussian voxelization process deviates from a strict mathematical definition. It appears they approximate Gaussians as isotropic cubes rather than treating them as anisotropic spheres precisely.

In the context of GaussianFormer, with its typically small Gaussians, this simplification might not lead to a significant performance degradation; it could be an acceptable practical compromise for their design. However, for GaussTR, which deals with larger and fewer Gaussians—particularly those used to represent broad, flat features like roads—such a simplification would definitely compromise performance.

I haven't yet had the opportunity to discuss it with the GaussianFormer authors. I hope my interpretation of their implementation is correct.

[Agito555]

Hi,

Regarding the comment about GaussianFormer's CUDA implementation, my analysis indicates that their Gaussian voxelization process deviates from a strict mathematical definition. It appears they approximate Gaussians as isotropic cubes rather than treating them as anisotropic spheres precisely.

In the context of GaussianFormer, with its typically small Gaussians, this simplification might not lead to a significant performance degradation; it could be an acceptable practical compromise for their design. However, for GaussTR, which deals with larger and fewer Gaussians—particularly those used to represent broad, flat features like roads—such a simplification would definitely compromise performance.

I haven't yet had the opportunity to discuss it with the GaussianFormer authors. I hope my interpretation of their implementation is correct.

Thank you to the author for their work, open-source code, and maintenance and support. I am a beginner in CUDA and Gaussian splatting, and I would like to ask why the author mentioned that in GaussianFormer ”they approximate Gaussians as isotropic cubes rather than treating them as anisotropic spheres precisely“. As far as I understand, whether it is an anisotropic sphere is related to S and R, and from what I have seen in the implementation of GaussianFormer so far, there doesn't seem to be any constraint on S and R? Or did I miss some key aspects?

[npurson]

Hi,
Regarding the comment about GaussianFormer's CUDA implementation, my analysis indicates that their Gaussian voxelization process deviates from a strict mathematical definition. It appears they approximate Gaussians as isotropic cubes rather than treating them as anisotropic spheres precisely.
In the context of GaussianFormer, with its typically small Gaussians, this simplification might not lead to a significant performance degradation; it could be an acceptable practical compromise for their design. However, for GaussTR, which deals with larger and fewer Gaussians—particularly those used to represent broad, flat features like roads—such a simplification would definitely compromise performance.
I haven't yet had the opportunity to discuss it with the GaussianFormer authors. I hope my interpretation of their implementation is correct.

Thank you to the author for their work, open-source code, and maintenance and support. I am a beginner in CUDA and Gaussian splatting, and I would like to ask why the author mentioned that in GaussianFormer ”they approximate Gaussians as isotropic cubes rather than treating them as anisotropic spheres precisely“. As far as I understand, whether it is an anisotropic sphere is related to S and R, and from what I have seen in the implementation of GaussianFormer so far, there doesn't seem to be any constraint on S and R? Or did I miss some key aspects?

I'm also a CUDA newbie and I didn't delve into their code. I draw the conclusion from the attached visualization results using their voxelization process, where all predicted anisotropic spheres turn into isotropic cubes.

Feel free to point out if I was doing wrong. ; )

[seamie6]

To me, that appears to be the 'scale multiplier' visualization. Similar to your code, this is a value to govern an area surrounding the Gaussian to model the decay of the Gaussian to reduce computation costs. This seems to be dependent on Gaussian covariance (scale multiplier is embedded into the CUDA ops, so I do not know where exactly it is used). Although I could be wrong.

Examining the forward CUDA code, it appears they do correctly model Gaussian decay as anisotropic ellipsoids, correctly including covariances.

[Agito555]

I'm glad to receive messages from both of you and to engage in this discussion.
I agree with Seamie's perspective; it seems that their code models the shapes as anisotropic ellipsoids.This is reflected in their code, where different voxel points get different values because of this modeling.
Their scale multiplier is used to multiply the scale to control the radii of each Gaussian's influence area.
However, I'm not quite sure about their visualization code, as it looks rather strange. :)

[seamie6]

Hi,
Regarding the comment about GaussianFormer's CUDA implementation, my analysis indicates that their Gaussian voxelization process deviates from a strict mathematical definition. It appears they approximate Gaussians as isotropic cubes rather than treating them as anisotropic spheres precisely.
In the context of GaussianFormer, with its typically small Gaussians, this simplification might not lead to a significant performance degradation; it could be an acceptable practical compromise for their design. However, for GaussTR, which deals with larger and fewer Gaussians—particularly those used to represent broad, flat features like roads—such a simplification would definitely compromise performance.
I haven't yet had the opportunity to discuss it with the GaussianFormer authors. I hope my interpretation of their implementation is correct.

Thank you to the author for their work, open-source code, and maintenance and support. I am a beginner in CUDA and Gaussian splatting, and I would like to ask why the author mentioned that in GaussianFormer ”they approximate Gaussians as isotropic cubes rather than treating them as anisotropic spheres precisely“. As far as I understand, whether it is an anisotropic sphere is related to S and R, and from what I have seen in the implementation of GaussianFormer so far, there doesn't seem to be any constraint on S and R? Or did I miss some key aspects?

I'm also a CUDA newbie and I didn't delve into their code. I draw the conclusion from the attached visualization results using their voxelization process, where all predicted anisotropic spheres turn into isotropic cubes.

Feel free to point out if I was doing wrong. ; )

I misread this as you took this image from their repo. But I see you said you generated it yourself. Could you share the code for generating this, specifically the size of the Gaussians, as I do think this could be related to the scale multiplier.

[npurson]

To me, that appears to be the 'scale multiplier' visualization. Similar to your code, this is a value to govern an area surrounding the Gaussian to model the decay of the Gaussian to reduce computation costs. This seems to be dependent on Gaussian covariance (scale multiplier is embedded into the CUDA ops, so I do not know where exactly it is used). Although I could be wrong.

Examining the forward CUDA code, it appears they do correctly model Gaussian decay as anisotropic ellipsoids, correctly including covariances.

Thank you for digging into the CUDA kernel to verify! Given these analysis, there might be bugs in my implementation. Unfortunately, the script used to generate that visualization was from an early exploratory phase, and I don't have it readily available to re-examine.

Due to my current workload on other projects, I won't have the bandwidth to reproduce it in the near future, but I truly appreciate you taking the time to investigate and post this clarification. It's a valuable correction for everyone following the issue.