Skip to content

polysemous_ht resets to 0 when loading from disk #2120

Description

@mhendrey

Summary

I run a hyperparameter sweep for an IVF,PQ index which provides optimal values for nprobe,ht. I then set those
parameters for the index and save it to disk. When I read from index, the nprobe is still the same, but the ht has
been reset to 0.

Platform

OS: Ubuntu 20.04.3 LTS

Faiss version: 1.7.1

Installed from: anaconda, pytorch channel

Running on:

  • CPU
  • GPU

Interface:

  • C++
  • Python

Reproduction instructions

import faiss

n = 10000
d = 64
nlist = 100
X = faiss.randn((n, d))

index = faiss.index_factory(d, f"IVF{nlist},PQ8")
print(f"{index.do_polysemous_training=:}")
print(f"{index.by_residual=:}")
index.train(X)
index.add(X)

params = faiss.ParameterSpace()
params.set_index_parameters(index, "nprobe=10,ht=32")
faiss.write_index(index, "testing.index")

index2 = faiss.read_index("testing.index")
assert index.nprobe == index2.nprobe, f"{index.nprobe=:}, {index2.nprobe=:}"
assert index.polysemous_ht == index2.polysemous_ht, f"{index.polysemous_ht=:}, {index2.polysemous_ht=:}"

"""
index.do_polysemous_training=True
index.by_residual=True
Traceback (most recent call last):
File "faiss_bug.py", line 21, in
assert index.polysemous_ht == index2.polysemous_ht,
AssertionError: index.polysemous_ht=32, index2.polysemous_ht=0
"""

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions