🕵️ Autonomous E-Commerce Competitor Intelligence Agent

An autonomous, multi-agent AI application built with LangGraph and Streamlit that helps you analyze the competitive landscape for any e-commerce product category.

The agent automatically finds competitor websites, scrapes product data, extracts structured intelligence, saves it to a database, and generates a comprehensive qualitative analysis report.

✨ Features

• Automated Competitor Discovery: Uses Firecrawl to search the web and identify top competitor URLs for a given product category.
• Intelligent Scraping & Extraction: Scrapes web pages into Markdown and uses a highly resilient LLM fallback chain to reliably extract structured product data (price, stock status, materials, etc.).
• Robust LLM Fallback Mechanism: Implements a transparent, sequential fallback chain (Groq → OpenRouter → OpenAI → Gemini) to ensure the agent never fails due to rate limits or API downtime.
• Data Persistence: Automatically stores all extracted competitor products into a local MySQL database for future querying and historical analysis.
• Qualitative Analysis: Uses Pandas for statistical number crunching and an LLM to generate a professional Markdown report detailing market gaps and pricing strategies.
• Interactive UI: Built with Streamlit for a clean, user-friendly dashboard.

🛠️ Architecture

The application is powered by a LangGraph state machine consisting of three main nodes:

1. Searcher Node (src/agents/searcher.py): Finds competitor URLs.
2. Scraper Node (src/agents/scraper.py): Extracts raw markdown and parses it into structured Pydantic models.
3. Analyst Node (src/agents/analyst.py): Saves data to MySQL, computes Pandas statistics, and writes the final report.

🚀 Getting Started

Prerequisites

• Python 3.9+
• MySQL Server running locally (or remotely)

1. Clone the Repository

git clone <your-repo-url>
cd ecom-com

2. Install Dependencies

It is recommended to use a virtual environment.

python -m venv venv
source venv/bin/activate  # On Windows use `venv\Scripts\activate`
pip install -r requirements.txt

3. Database Setup

Ensure you have a MySQL server running. Create a database named ecom-search (or whatever you configure in your environment). The application will automatically create the necessary tables on startup.

4. Environment Configuration

Create a .env file in the root directory and add your API keys and Database credentials.

Note: OpenAI is mandatory for the fallback chain to guarantee extraction reliability.

# Database Configuration
DB_HOST=localhost
DB_PORT=3306
DB_NAME=ecom-search
DB_USERNAME=root
DB_PASSWORD=your_db_password_here

# LLM Providers (OpenAI is required, others are optional fallbacks)
OPENAI_API_KEY=your_openai_api_key_here
GEMINI_API_KEY=your_gemini_api_key_here
GROQ_API_KEY=your_groq_api_key_here
OPEN_ROUTER_API_KEY=your_open_router_api_key_here

# Scraping API
FIRECRAWL_API_KEY=your_firecrawl_api_key_here

5. Run the Application

Start the Streamlit dashboard:

streamlit run app.py

🧠 LLM Fallback Strategy

Web scraping requires processing massive amounts of tokens, which can quickly trigger API rate limits. To ensure stability, this app uses a custom fallback chain located in src/llm_utils.py.

The sequence is:

1. Groq (llama-3.3-70b-versatile): Fast and free, but strict rate limits.
2. OpenRouter: Secondary cost-effective fallback.
3. OpenAI (gpt-4o-mini / gpt-4o): The highly reliable core engine.
4. Gemini (gemini-2.0-flash): Final safety net.

If an LLM hits a rate limit (429), it immediately falls back to the next provider and logs the failure transparently.