Skip to content

fix(ingestion): contradiction LLM check fires for every near-match at sim>0.30 — too aggressive and expensive #69

Description

@laxmanclo

extractor.py _check_dedup() fires a single LLM call for every ingested fact that has any stored fact with sim>0.30. With 100+ facts stored this is O(N) LLM calls per ingestion. Fix: (1) skip the LLM check when the earlier high-sim dedup branch (>0.90) already handled it; (2) raise the threshold from 0.30 to ~0.50; (3) optionally batch multiple new facts into one LLM call.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingperformancePerformance improvements

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions