index out of range error when corpus size is in thousands

[manas007]

Hi, thank you for your work on this.
I am noticing that I run into index out of range error when my corpus is around 1k-2k documents

i have precomputed embeddings stored in my data.

import umap
document_vectors = np.stack(df['EMBEDDINGS'].values)
document_map = umap.UMAP(metric='cosine').fit_transform(document_vectors)

from toponymy import ToponymyClusterer
clusterer = ToponymyClusterer(min_clusters=6)
clusterer.fit(clusterable_vectors=document_map, embedding_vectors=document_vectors)
for i, layer in enumerate(clusterer.cluster_layers_):
    print(f'{len(np.unique(layer.cluster_labels))-1} clusters in layer {i}')

Output:
111 clusters in layer 0
36 clusters in layer 1
9 clusters in layer 2

so far, all good.

from sentence_transformers import SentenceTransformer
from toponymy import Toponymy, KeyphraseBuilder
from toponymy.llm_wrappers import HuggingFace

embedding_model = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B")
llm = HuggingFace("Qwen/Qwen2.5-1.5B-Instruct")

text = df['TITLE_TEXT'].values

topic_model = Toponymy(
    llm_wrapper=llm,
    text_embedding_model=embedding_model,
    clusterer=clusterer,
)
topic_model.fit(text, document_vectors, document_map)

topic_names = topic_model.topic_names_

IndexError: list index out of range
File , line 8
1 text = df['TITLE_TEXT'].values
3 topic_model = Toponymy(
4 llm_wrapper=llm,
5 text_embedding_model=embedding_model,
6 clusterer=clusterer,
7 )
----> 8 topic_model.fit(text, document_vectors, document_map)
10 topic_names = topic_model.topic_names_
11 topics_per_document = [cluster_layer.topic_name_vector for cluster_layer in topic_model.cluster_layers_]
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-bf30105a-4657-4047-8f9f-0e0ae989b858/lib/python3.11/site-packages/toponymy/cluster_layer.py:229, in ClusterLayer._update_topic_names(self, new_topic_names, topic_indices)
225 """
226 Update the topic names for the specified indices.
227 """
228 for i, topic_index in enumerate(topic_indices):
--> 229 self.topic_names[topic_index] = new_topic_names[i]

I am running this on databricks with cluster config as below:
databricks runtime : 15.4 LTS (includes Apache Spark 3.5.0, Scala 2.12)
nodetype Standard_DS3_v2

[lmcinnes]

Thanks for the report. We are definitely still working out all the kinks and corner cases. This feels like something I believe we saw before at one point, but I thought we fixed. Is the dataset shareable so I can try to reproduce? I do understand that that is not always the case...

[manas007]

@lmcinnes thank you for your quick response. apologies, the data is not shareable, however, I will try to reproduce the error on other open source dataset and see if i run into the same error

[arcrystal]

Hi @lmcinnes @manas007 I am seeing the same issue with a dataset of size 270k. With some crude debugging it looks like are new_topic_names with length 0 that are being indexed. I got around it with try/except block (if there aren't new topic names, it seems feasible to use the old ones).

try: self.topic_names[topic_index] = new_topic_names[i] 
except IndexError: continue

I'm not sure if this "fix" resulted in or is related to the following new error, or if is unrelated, but later during disambiguation prompt construction I get this error:

Traceback (most recent call last):
  File "src/topic_modeling_toponymy.py", line 159, in
    main()
  File "src/topic_modeling_toponymy.py", line 130, in main
    toponymy_model = toponymy_model.fit(docs, embeds, doc_map)
  File "toponymy/toponymy.py", line 271, in fit
    self.topic_names_[i] = layer.name_topics(
  File "toponymy/cluster_layer.py", line 440, in name_topics
    self.disambiguate_topics(
  File "toponymy/cluster_layer.py", line 277, in disambiguate_topics
    self._make_disambiguation_prompts(
  File "toponymy/cluster_layer.py", line 186, in _make_disambiguation_prompts
    self.disambiguation_prompts = [
  File "toponymy/cluster_layer.py", line 187, in
    distinguish_topic_names_prompt(
  File "toponymy/prompt_construction.py", line 178, in distinguish_topic_names_prompt
    unique_topic_names = list(dict.fromkeys(attempted_topic_names))
TypeError: unhashable type: 'list'

I'm seeing that attempted_topics_names contains some elements that are lists.

Just wanted to add to the thread and hope it helps for debugging. Thanks for you work on this!

[jc-healy]

Thanks for this. Did you want to submit a small PR with your try/except fix? It does seem reasonable to throw an error and fail gracefully back to previous topic names in such cases.

[arcrystal]

Thanks for this. Did you want to submit a small PR with your try/except fix? It does seem reasonable to throw an error and fail gracefully back to previous topic names in such cases.

Just submitted a pull request, but added a caveat in the comment because I wonder whether the issue is further upstream when Toponymy creates the new_topic_names and fixing that would take care of this issue.