Skip to content

Enforce model-level structured output in agent enrichment flow #186

Description

@ydvbobby

Issue

The agent enrichment pipeline currently relies on prompt instructions to produce JSON, then
parses and validates the raw text response afterward. This is fragile and makes structured
extraction dependent on prompt compliance rather than model-level enforcement.

Current behavior:

  • BaseAgent calls llm.invoke(prompt) directly
  • raw text is parsed with _parse_json_value / _parse_json_object
  • parsed data is validated afterward with TypeAdapter

This means:

  • malformed JSON can still be returned by the model
  • prompt wording is carrying too much responsibility
  • provider behavior is inconsistent across the fallback chain
  • agent logic contains repeated parse/recovery code that should not be in the happy path

Goal

Move structured extraction to model-level enforcement using .with_structured_output(...)
where supported, so agent tasks receive validated Python objects instead of raw JSON text.

Scope

  • audit provider support across the current FreeTierLLM fallback chain
  • add a structured-output wrapper at the LLM factory/base-agent layer
  • map each enrichment task to an explicit Pydantic schema
  • update agent flows to use structured invocation instead of manual JSON parsing
  • define fallback behavior for providers that do not support structured output
  • add tests for supported, unsupported, and failover scenarios

Suggested approach

  1. Add a schema-aware LLM accessor in the config/base-agent layer.
  2. Implement a _run_structured(prompt, schema) helper in BaseAgent.
  3. Start with one low-risk enrichment task such as SocialMedia or Contact.
  4. Expand to Education, PoliticalBackground, FamilyMember, and CrimeRecord.
  5. Retain prompt-based JSON parsing only as a controlled fallback if required.

Metadata

Metadata

Labels

No labels
No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions