init-plan.md

This is an excellent project choice. Building an autonomous competitor intelligence agent moves you away from simple RAG chatbots and straight into production-grade multi-agent orchestration.

In Python, the engineering challenge isn’t just making the LLM smart; it’s building a robust state machine around the LLM that handles unpredictable data, rate limits, and failure modes seamlessly.

Here is a deep architectural dive and the context you need to build this successfully in Python.

---

1. System Architecture & The "State" Shape

To keep your agents from drifting into endless loops or forgetting what they've scraped, you need to manage the State centrally. If you use a framework like LangGraph (which is ideal for this), the state is passed from node to node as a single Python dictionary or object.

Here is how you should structure the shared state object:

from typing import List, Dict, Any
from pydantic import BaseModel, Field

class CompetitorProduct(BaseModel):
    competitor_name: str
    product_name: str
    price: float
    currency: str
    in_stock: bool
    material: str = Field(description="Materials extracted, e.g., Full-grain leather")
    url: str

class AgentState(BaseModel):
    category: str                   # e.g., "Men's Oxford Shoes"
    competitor_urls: List[str] = [] # Discovered by Agent 1
    scraped_raw_data: List[str] = []# Extracted page markdowns
    extracted_products: List[CompetitorProduct] = [] # Structured data from Agent 2
    analysis_report: str = ""       # Final output from Agent 3
    errors: List[str] = []          # Error tracking for self-healing loops

---

2. Deep Dive: The 3 Python Agents

Agent 1: The Searcher (Discovery Phase)

The Problem: Traditional Google search returns generic consumer articles rather than actual e-commerce category pages.

• The Tool: Use a developer-focused search engine like Exa API or Firecrawl Search. Exa allows you to search based on semantic meaning and filter for domain types (e.g., forcing results to include only /collections/ or /products/ keywords).
• Python Logic: Your prompt should take the category "Men's Oxford Shoes" and instruct the search engine to look for specific top e-commerce players or direct marketplace links.

Agent 2: The Dynamic Scraper (The Extraction Engine)

The Problem: E-commerce sites use dynamic JavaScript, heavy anti-bot protections, and wildly changing HTML hierarchies. Sending pure raw HTML to an LLM will break your context window limit and cost a fortune.

• The Modern Stack Choice: Instead of writing custom raw Playwright code, use specialized libraries built for LLM extraction like ScrapeGraphAI or Crawl4AI / Firecrawl. These tools handle turning bloated HTML into super-lean, compressed Markdown.
• The Strategy: Pass the Markdown to a fast, cost-efficient model (like gpt-4o-mini or gemini-1.5-flash) utilizing Structured Outputs via Pydantic.

# The extraction structure you feed to the LLM via Instructor or LangChain
class CategoryPageExtraction(BaseModel):
    items: List[CompetitorProduct]

# If a scrape fails due to an anti-bot wall, Agent 2 writes the error 
# to the state, allowing the loop to retry using a different proxy profile.

Agent 3: The Analyst (Data Synthesis)

The Problem: Raw data doesn't provide business value. This agent acts as the brain that turns data into action.

• Python Logic: Do not use the LLM to calculate averages or match strings—LLMs are bad at pure math. Instead, write native Python functions (tools) using Pandas to compute price distributions, standard deviations, and catalog overlapping.
• The LLM's Role: Pass the Pandas statistical summary back to the LLM. The LLM's job is purely qualitative context: identifying why a competitor is priced higher (e.g., "Competitor X uses Goodyear-welted construction, justifying a 40% price premium over your cemented soles") and drafting the Markdown report.

---

3. Designing a "Self-Healing" Execution Flow

In production, scraping fails roughly 15% of the time due to popups, cookie consent banners, or bad network calls. Your Python application should be designed as a graph with a conditional fallback loop:

                  [ Agent 1: Search ]
                           │
                           ▼
                  [ Agent 2: Scrape ] ◄────────┐
                           │                   │
                  Did Extraction Succeed?      │ (If Validation Fails:
                   /               \           │  Retry with clean HTML
                 YES                NO ────────┘  or altered proxy)
                  │
                  ▼
              [ Pandas Math Processing ]
                           │
                           ▼
                  [ Agent 3: Analyst ]

---

4. Immediate Libraries to Install

To start prototyping this in Python, open your environment and pull these down:

pip install pydantic instructor openai pandas
# If choosing open-source local-first graph engines:
pip install langgraph scrapegraphai 
# Or if choosing managed API-first scraping:
pip install firecrawl-py