Support for CRC-64/NVME

[normj]

Amazon S3 recently launched new data integrity checks when uploading and download objects. https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html. For context I'm a maintainer of the AWS .NET SDK.

It supports a collection of different hashing algorithms but the default is CRC-64/NVME which the System.IO.Hashing package does not support. The default means if an object was uploaded without a checksum, using an older SDK, S3 automatically generates a CRC-64/NVME checksum at the service level. Objects can also get a CRC-64/NVME checksum if they use another language that does support CRC-64/NVME like Go. This puts .NET at a disadvantage because users that are downloading with .NET objects that have a CRC-64/NVME checksum have the data integrity check skipped.

Curious what the .NET team's appetite is for adding CRC-64/NVME support in System.IO.Hashing. It has a CRC64 implementation for ECMA-182 today. Depending how this GitHub issue API request is handled that could solve the S3 need. #78063

[jeffhandley]

Hi @normj - With Vectorize custom polynomial CRC-32 and CRC-64 (#124832), we're now finished with the related feature work on this.

The scalar implementation was included in .NET 11 Preview 2 and the vectorized implementation will be in .NET 11 Preview 3.

Please let us know if you have any feedback or issues--best to just file a new issue if you do, tagging @bartonjs and me please. Otherwise, we'd also appreciate a response here if you successfully integrate it.

Thanks!

[normj]

Thanks @jeffhandley on the update. It is on my list of .NET 11 features to test out with the AWS SDK along with the new async changes. I'll try and get to trying out the new CRC-64 support with the preview in the next couple weeks. I'll respond back whether I was successful or not.