Skip to content

architecture.md

Latest commit

 

History

History
215 lines (163 loc) · 12.9 KB

architecture.md

File metadata and controls

215 lines (163 loc) · 12.9 KB
Raw

TSG Builder Architecture

This document describes the internal architecture and design details of TSG Builder.

Multi-Stage Pipeline

TSG Builder uses a three-stage pipeline to separate concerns and improve reliability:

┌─────────────────┐     
│  Raw Notes      │     
│  (input.txt)    │     
└────────┬────────┘     
         │
         ▼
┌──────────────────────────────────────────────────────────────┐
│  PRE-FLIGHT: PII CHECK                                      │
│  - Azure AI Language PII API (via Foundry AI Services)      │
│  - Scans notes for emails, phone numbers, credentials, etc. │
│  - Fail-closed: blocks generation if PII found OR on error  │
│  → Pass: proceed to pipeline │ Fail: user edits or redacts  │
└────────────────────────────┬─────────────────────────────────┘
         │
         ▼
┌────────────────────────────────────────────────────────────────────┐
│                    TSG PIPELINE                                    │
│  ┌──────────────────────────────────────────────────────────────┐ │
│  │  Stage 1: RESEARCH (has tools)                               │ │
│  │  - Microsoft Learn MCP → official docs, APIs, limits         │ │
│  │  - Web Search → GitHub issues, community solutions           │ │
│  │  → Output: Structured research report with URLs              │ │
│  └──────────────────────────────────────────────────────────────┘ │
│                              │                                     │
│                              ▼                                     │
│  ┌──────────────────────────────────────────────────────────────┐ │
│  │  Stage 2: WRITE (no tools)                                   │ │
│  │  - Uses ONLY notes + research report                         │ │
│  │  - Follows TSG template exactly                              │ │
│  │  → Output: Draft TSG + {{MISSING::...}} placeholders         │ │
│  └──────────────────────────────────────────────────────────────┘ │
│                              │                                     │
│                              ▼                                     │
│  ┌──────────────────────────────────────────────────────────────┐ │
│  │  Stage 3: REVIEW (no tools)                                  │ │
│  │  - Structure validation (headings, markers)                  │ │
│  │  - Fact-check against research (soft warnings)               │ │
│  │  - Auto-fix simple issues, retry if needed                   │ │
│  │  → Output: Validated TSG + review notes                      │ │
│  └──────────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────┘
         │
         ▼
┌──────────────────────────────────────────┐
│  Structured TSG (markdown)               │
│  + Follow-up Questions OR NO_MISSING     │
│  + Review warnings (if any)              │
└──────────────────────────────────────────┘

Stage Details

Stage 1: Research

The Research agent has access to external tools and is responsible for gathering information:

  • Microsoft Learn MCP searches for:

    • Azure service documentation
    • Known limitations and workarounds
    • Configuration guides

  • Web Search (uses Bing under the hood) for:

    • GitHub discussions and issues
    • Community workarounds
    • Stack Overflow solutions

Output: A structured research report with URLs and key findings.

Why tools here? Research needs external data. By isolating tool use to this stage, we ensure research actually happens and can track what sources were used.

Stage 2: Write

The Writer agent has no tool access and works only from:

  • Original user notes
  • Research report from Stage 1

Output: Draft TSG following the template, with {{MISSING::...}} placeholders for gaps.

Why no tools? Prevents the writer from making ad-hoc searches that could introduce unverified information. All claims must trace back to the research report.

Stage 3: Review

The Reviewer agent validates the draft:

  • Structure validation: All required sections and markers present
  • Fact-checking: Claims match the research report (flags potential hallucinations)
  • Auto-correction: Fixes simple issues automatically
  • Retry logic: Up to 2 retries if validation fails

Output: Validated TSG + review notes/warnings.

Deterministic diagnosis-line guarantee: The mandatory line Don't Remove This Text: Results of the Diagnosis should be attached in the Case notes/ICM. is enforced by a model-independent post-process (ensure_required_diagnosis_line()) after the review loop, so it is present in the Diagnosis section of every generated TSG that has a # **Diagnosis** heading. (If the heading itself is missing — a structural failure the review retry loop owns — the post-process defers rather than masking it.)

Offline eval harness: evals/ holds a deterministic, no-network quality guardrail (make eval) that scores captured TSG outputs for publication-blocking failure modes — template compliance, MISSING hygiene, code-token fidelity, and source-attribution leakage. Scorer failure modes share the quality_taxonomy.py vocabulary with in-product tsg_feedback telemetry so offline and real-world signals are comparable. See evals/README.md.

Output Format

The pipeline outputs structured markers for parsing:

<!-- TSG_BEGIN -->
[TSG markdown content]
<!-- TSG_END -->

<!-- QUESTIONS_BEGIN -->
[Follow-up questions or NO_MISSING]
<!-- QUESTIONS_END -->

Missing Information Placeholders

When information is missing, the agent inserts placeholders:

{{MISSING::<SECTION>::<HINT>}}

For example:

{{MISSING::Root Cause::Specify the exact error code from the customer's logs}}

Iteration Flow

When the TSG has missing information:

  1. Agent inserts {{MISSING::...}} placeholders in the TSG
  2. Agent generates targeted follow-up questions
  3. User provides answers
  4. Pipeline regenerates with the new information
  5. Repeat until NO_MISSING is returned

File Structure

File Purpose
pipeline.py Multi-stage pipeline orchestration, error classification
tsg_constants.py TSG template, agent instructions, and stage prompts
pii_check.py PII detection via Azure AI Language API (pre-flight gate)
error_utils.py Shared Azure SDK error classification utilities
telemetry.py Anonymous usage telemetry (see docs/telemetry.md)
version.py Single source of truth for version, GitHub URL, and TSG signature
web_app.py Flask web UI + agent creation
build_exe.py PyInstaller build script (bundles templates/, static/)
validate_setup.py Validate environment configuration (CLI troubleshooting)
delete_agents.py Delete agents from Azure (used by make clean DELETE_AGENTS=1)
Makefile Common operations (setup, ui, test, build, clean)
templates/index.html Web UI HTML structure
static/css/styles.css Web UI styles
static/js/main.js Core application logic (streaming, TSG display, images, PII modal)
static/js/setup.js Setup modal functionality
.agent_ids.json Stores agent IDs after creation
tests/ Pytest test suite
tests/conftest.py Shared fixtures and test utilities
docs/archive/error-handling-plan.md Error handling implementation plan (archived)
docs/archive/pii-detection-plan.md PII detection implementation plan (archived)

Design Decisions

Supported Model Policy

TSG Builder supports non-chat deployments of gpt-5.1, gpt-5.2, gpt-5.4, and gpt-5.5.

This policy is intentionally conservative:

  1. Agent tool support is model-specific — The pipeline depends on Azure AI Foundry Agent Service tools, image input, and structured review output.
  2. Sibling variants are separate models — -chat, -mini, -nano, -pro, and -codex variants stay blocked until separately validated.

The setup UI asks for a deployment name because users choose deployment names. Validation checks the deployment's underlying model and blocks unsupported models before agents are created.

Why Three Separate Agents?

  1. Separation of concerns — Each agent has a focused role with appropriate capabilities
  2. Tool isolation — Only the researcher has tool access, preventing uncontrolled external calls
  3. Traceability — Clear audit trail of what was researched vs. what was generated
  4. Retry granularity — Can retry individual stages without re-running the whole pipeline

Why Remove Tools from Writer?

Early experiments showed that giving the writer tool access led to:

  • Ad-hoc searches that contradicted the research report
  • Hallucinated URLs (the model would "search" but make up results)
  • Inconsistent quality

By forcing the writer to use only the research report, outputs are more consistent and traceable.

Why a PII Pre-Flight Check?

TSG Builder sends user-provided notes to external services (Foundry Agents, Bing search). A pre-flight PII check prevents customer-identifiable information from leaking:

  • Fail-closed — If the PII API is unreachable or errors, generation is blocked. Input cannot be sent to external services without PII clearance.
  • Uses existing Foundry resource — The PII check uses the AI Services endpoint built into the user's Foundry resource (derived from PROJECT_ENDPOINT). No additional setup or resources needed.
  • No bypass — There is no "proceed anyway" option. Users must edit their notes or accept the API's automatic redaction before generation can continue.
  • Curated categories — Only categories relevant to support scenarios are detected (emails, phone numbers, IP addresses, credentials, etc.). Organization is intentionally excluded to avoid false positives on Azure service names.
  • Defense-in-depth — The frontend checks before sending, and the backend re-checks at the /api/generate/stream and /api/answer/stream endpoints to prevent bypass.

Session Persistence

In-progress work can be saved and resumed across server restarts. Sessions are stored as JSON files under .sessions/ in the app directory (alongside .agent_ids.json), one file per session keyed by a session_id (UUID) that is distinct from the pipeline thread_id.

  • Auto-save — every successful pipeline run persists the session (notes, images, TSG, research, review, and iteration state), so completed work is never silently lost. Best-effort: persistence failures never block the pipeline.
  • Explicit save — the 💾 Save button persists the current client state (notes + images) even before generation, so input can be saved and resumed later.
  • Merge-aware — a follow-up save that doesn't resend images preserves the originals; a user-set label and the original creation time are preserved across auto-saves.
  • Restore — loading a session repopulates both client state and the in-memory sessions map so follow-up iteration continues seamlessly.
  • Local trust boundary — session files contain raw notes and images. They live only on the user's disk (same boundary as .env/.agent_ids.json), are gitignored, and never enter telemetry. Persisted-session REST operations live under /api/sessions (plural), distinct from the in-memory /api/session/<thread_id> cleanup endpoint.