InsightEngine deduplication can drop distinct URL-less results

[3em0]

Summary

InsightEngine can silently drop distinct search results when the results do not have a URL and their first 100 characters are identical.

The affected deduplication code uses only a truncated content prefix as the fallback identifier:

identifier = result.url if result.url else result.title_or_content[:100]

Affected location:

• InsightEngine/agent.py, _deduplicate_results

Impact

For URL-less records, two different comments/documents can be treated as duplicates even when the meaningful information differs after the first 100 characters. The later result is removed before it reaches downstream summary generation, sentiment analysis, search history, or final report output.

This can lead to incomplete or misleading InsightEngine reports. A realistic case is two comments about the same event sharing a long common prefix, while one contains mitigation guidance and the other contains contradictory or incomplete information after character 100.

Reproduction

Create two result-like objects with:

• url = None
• identical title_or_content[:100]
• different full title_or_content after character 100

Then pass them to _deduplicate_results.

Observed behavior:

• input count: 2
• output count: 1
• the second distinct result is silently dropped

Minimal reproduction logic:

seen = set()
unique_results = []

for result in results:
    identifier = result.url if result.url else result.title_or_content[:100]
    if identifier not in seen:
        seen.add(identifier)
        unique_results.append(result)

With two URL-less records sharing the same first 100 characters, identifier is equal even though the full content differs.

Expected Behavior

Distinct URL-less results should not be deduplicated solely by a truncated 100-character prefix. The deduplication key should preserve enough identity information to distinguish records whose full content, source table, author, timestamp, platform, or other stable fields differ.

Affected Paths

URL-less results are a real code path. For example, comment search results are built without a URL in:

• InsightEngine/tools/search.py, get_comments_for_topic

The deduplicated results are then converted into user/LLM-facing title, content, and raw_content fields in:

• InsightEngine/agent.py, initial search flow
• InsightEngine/agent.py, reflection search flow

Environment

• Repository: 666ghj/BettaFish
• Version shown in README: v1.2.1
• Commit checked locally: 53f60e8f4aa038a59ab5e02525634c92d6c849b5