Adding HannoyTransformer

[amalia-k510]

This PR adds HannoyTransformer as proposed in #123, wrapping hannoy (LMDB-backed storage) into the sklearn-ann transformer interface. The approach follows the same pattern as AnnoyTransformer and others.

A few things to note:

• hannoy's Python bindings don't expose by_item yet (it exists in the Rust code in reader.rs but PyReader only has by_vec). A PR upstream was created to add it. Until that is approved, fit_transform stores the training data and re-queries it using by_vec instead of the faster by_item path. I plan to switch to by_item once it's available.

• There's a known issue where multiple Database instances in the same process silently share the first LMDB environment.

• I also opened an issue on hannoy requesting batch insert/query APIs to avoid the per-vector Python to Rust loop overhead.

Two questions: which metrics do we want to support? Right now it's only euclidean, but hannoy also has hamming, sqeuclidean, cosine, and manhattan. Also hannoy offers binary quantized variants of consine, euclidean, and manhattan. Would we want to also use those?

[flying-sheep]

There's a known issue where multiple Database instances in the same process silently share the first LMDB environment.

Do you mean a hannoy issue (if so, link plz) or in your code?

which metrics do we want to support?

All of them of course! Your code should not know which ones exist (except for the default): instead of accepting a string, just use the Metric enum as a type and pass that through to the upstream API.

[flying-sheep]

why is that?

[amalia-k510]

When I checked's hannoy's euclidean distance function, it seems to returns the squared distance it computes but never takes the square root. So it returns a squared Euclidean distance, not the actual Euclidean distance. You can see it in the scalar fallback at src/spaces/simple.rs:49-51:
u.iter().zip(v.iter()).map(|(u, v)| (u - v) * (u - v)).sum()
I believe that this is intentional on hannoy's side. It seems more of a free performance win internally; however, for us, it means hannoy returns a squared Euclidean instead the actual Euclidean distance. The np.sqrt here finishes the computation so the output matches what scipy and KNeighborsTransformer expect.

[ilan-gold]

https://github.com/nnethercott/hannoy/commits/main/ Both Phil and my PRs have been merged - this could be developed against main to use by_item and add_items

[flying-sheep]

yeah, if it all works as expected, we can ask the maintainer to cut a release in nnethercott/hannoy#127

[flying-sheep]

Please add this transformer to test_kneighbors/test_common.py

[flying-sheep]

Will hannoy only ever output float32?

[amalia-k510]

Yes, it seems to be the case, when I was looking through the code. When looking at the hannoy's distance the function returns as -> f32 (src/distance/mod.rs:41), so every metric returns f32 distances. The Python bindings also do the same thing: by_vec → Vec<(ItemId, f32)> and by_item → Option<Vec<(ItemId, f32)>> (src/python.rs:402 and L415-L420).

[flying-sheep]

Oof that looks slow. Are these APIs enough to avoid for loops? nnethercott/hannoy#127

If not, we should file another issue.

[amalia-k510]

From what I understand, it is not yet the case. I am having a bit of a hard time examining Rust code but, it seems like the batch query is still missing, so the loops here are needed for now.

In the issue #127 that I opened up, I mentioned three things: by_item, batch insert, and batch query. The first two were resolved and merged from what I am understanding; however, the read side has no batch method. So _transform still makes one call into Rust per row, which is the slow part you flagged. I will add another comment on this issue #127 to bring up to the attention.

[amalia-k510]

There's a known issue where multiple Database instances in the same process silently share the first LMDB environment.

Do you mean a hannoy issue (if so, link plz) or in your code?

This is on hannoy's side, not ours. hannoy stores its LMDB environment in a global OnceCell at src/python.rs:18, initialised via get_or_try_init in PyDatabase::new at src/python.rs:107-111. The first Database created in a process sets the path and env size for the whole process; every attempt to create a database after that reuses the same env and silently ignores its own path and env_size arguments.
To work around it, I give every fit() its own u16 index prefix (self._index_ = next(_index_counter) % 2**16) so different transformers write to separate keyspaces inside the shared env and don't overwrite each other.

[flying-sheep]

looks good, I’ll do a batch query PR for hannoy, and if the once author does a release, we can finish this up

[flying-sheep]

Can we make this more helpful? What can a user do to avoid this issue?