Add SVE low-dimensional nearest fast path and spherical SuperKMeans support

[marcelo-cjl]

Summary

This PR adds two related changes for the SCANN coarse quantizer / PQ encoding path on ARM SVE: a low-dimensional nearest-neighbor fast path and spherical SuperKMeans support.

1. Low-dimensional L2sqr nearest fast path (d in {2,4,8})

fvec_L2sqr_ny_nearest<ARM_SVE> previously wrote all ny distances to the scratch buffer and then performed a separate linear scan. For PQ encoding (compute_1_code) each call has d = dsub and ny = ksub, making this the hot path. The new path maps one centroid to each SVE lane, gathers its D components, and keeps lane-local minimum distances and ids across batches. A predicated tail handles non-multiple ny values, equal distances preserve the first index, and only one final horizontal reduction is needed. The scratch buffer is not written, matching the x86 AVX2/AVX512 D2/D4/D8 implementations.

ProductQuantizer::compute_code switches to AVAILABLE_SIMD_LEVELS_A1 so the ARM_SVE implementation is reachable from PQ encoding (A0 does not include ARM_SVE).

2. SuperKMeans spherical (inner-product) support

SuperKMeans previously only supported L2 clustering; the header explicitly excluded IP/cosine. With cp.spherical=true:

• TrainState::R becomes std::unique_ptr<VectorTransform>; power-of-two d uses the fast HadamardRotation, L2 keeps RandomRotationMatrix.
• Forgy initialization produces unit centroids before the first assignment. Each iteration computes and splits centroids first, then applies spherical centroid post-processing once, matching the Clustering flow.
• HadamardRotation::reverse_transform fills the missing inverse.
• ClusteringParameters::use_super_kmeans (default false) lets Level1Quantizer::train_q1 route coarse quantizer training through SuperKMeans when explicitly enabled; the Python type stubs expose the new option.
• block_l2<ARM_SVE> completes the SuperKMeans SIMD kernels for the ADSampling pruning loop.

Tests

• Low-dim nearest across SIMD levels: d in {2,4,8} and ny around common SIMD widths through 257, including non-multiple tails and first-index tie behavior. The dispatch test source is included in DD builds.
• Spherical objective close to vanilla spherical Clustering; unit-norm centroids.
• use_super_kmeans field inherited from ClusteringParameters.

Benchmark

qwen 4096-dim, IP, nlist=1024, sub_dim=4, 1 thread, 50k rows:

stage	baseline	optimized	change
Train	67.73s	24.36s	-64%
Add	10.86s	7.17s	-34%
Build	78.59s	31.52s	-60%

Train speedup comes from SuperKMeans spherical coarse quantizer training; Add speedup comes from the low-dim SVE nearest fast path in PQ encoding. recall@10 unchanged vs Clustering baseline (diff <= 0.0002).

Notes

This change targets ARM SVE. The x86 (AVX2/AVX512) implementations already have equivalent low-dim nearest specializations.

[meta-cla]

Hi @marcelo-cjl!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

[alibeklfc]

Hi, @marcelo-cjl. Thank you for your PR. Can you please sign Contributor License Agreement?
Is it ready for review? I see [WIP] in the title

[meta-cla]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

[marcelo-cjl]

Hi @alibeklfc, thanks for the reminder. The CLA has been signed, and the Meta CLA check is now passing. The PR is ready for review; the [WIP] marker has been removed, and the description has been updated to reflect the current implementation. Thanks!

[meta-codesync]

@alibeklfc has imported this pull request. If you are a Meta employee, you can view this in D117563661.