process employs a two-stage approach combining embedding-based clustering with LLM-based de-duplication to efficiently handle large knowledge graphs.
In the deduplicate() function, the role of clustering (node_clusters/edge_clusters) is as follows:
Divide all nodes/edges into several groups;
Submit each group as an independent task to the thread pool (to enable parallel batch processing);
Call deduplicate_cluster internally within each batch.
However, inside deduplicate_cluster, there is no logic to restrict retrieval or comparison to the current cluster.
All deduplication decisions are based on relevant items retrieved from the full dataset.
Therefore, clustering here is essentially a task partitioning mechanism rather than a semantic boundary constraint.
The implementation of get_relevant_items(query, top_k=16, type) explicitly uses:self.node_bm25:
constructed from all nodes;
self.node_embeddings: embeddings of all nodes;
Ultimately, it returns the global top-16 most relevant items, regardless of which cluster they belong to.
Even when cluster A is being processed, items from cluster B may still be retrieved as relevant_items.
This means the LLM may be required to determine whether two items from different clusters are duplicates — a scenario that clustering was originally intended to prevent.
In the deduplicate() function, the role of clustering (node_clusters/edge_clusters) is as follows:
However, inside deduplicate_cluster, there is no logic to restrict retrieval or comparison to the current cluster.
All deduplication decisions are based on relevant items retrieved from the full dataset.
Therefore, clustering here is essentially a task partitioning mechanism rather than a semantic boundary constraint.
The implementation of get_relevant_items(query, top_k=16, type) explicitly uses:self.node_bm25:
Ultimately, it returns the global top-16 most relevant items, regardless of which cluster they belong to.
Even when cluster A is being processed, items from cluster B may still be retrieved as relevant_items.
This means the LLM may be required to determine whether two items from different clusters are duplicates — a scenario that clustering was originally intended to prevent.