Skip to content

fix(llma): trace summarization missing $ai_evaluation event type #53532

@mango766

Description

@mango766

Bug

The trace summarization pipeline (posthog/temporal/llm_analytics/trace_summarization/) is missing $ai_evaluation from its AI event type lists:

  1. constants.pyAI_EVENT_TYPES tuple is missing $ai_evaluation
  2. sampling.py — Two hardcoded WHERE event IN (...) SQL clauses are missing $ai_evaluation

This means any $ai_evaluation events within a trace are silently excluded from:

  • Trace fetching (via queries.py which uses the AI_EVENT_TYPES constant)
  • Trace sampling (via the hardcoded SQL in sampling.py)

The canonical Node.js list in nodejs/src/ingestion/ai/process-ai-event.ts includes $ai_evaluation, and the event type is used across the codebase (schema, taxonomy, usage reports, etc.).

Impact

Traces that contain evaluation events will have incomplete summaries — the evaluation results are dropped before the LLM summarizer sees them.

Expected behavior

$ai_evaluation should be included alongside the other AI event types in both the constants tuple and the sampling SQL queries.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions