LLM-Based Routing Architecture

Purpose: Document the intelligent LLM-based routing system that eliminates hardcoded keyword patterns

Date: 2025-01-11

Related Commits:

0c99ba3 - LLM parser for intelligent routing
2329373 - Order status routing (deprecated approach)

Overview

The unified interactive CLI uses an LLM-based classification system to intelligently route user requests. This eliminates the need for hardcoded special cases and keyword patterns.

Core Principle

Let the LLM understand context, not just pattern-match keywords.

Instead of maintaining lists of phrases like:

# ❌ OLD APPROACH (doesn't scale)
if "any open orders" in input:
    show_orders()
elif "any positions" in input:
    show_positions()
# What about: "WHO ticker", "WHAT ticker", "WHEN ticker"???

We use the LLM to classify intent:

# ✅ NEW APPROACH (scalable)
request = await parser.parse(user_input)
if request.request_type == "status_query":
    route_to_status_handler()
elif request.request_type == "trade":
    route_to_trade_handler()

Architecture

Request Classification Flow

User Input: "any open orders?"
    ↓
LLM Parser (GPT-4o-mini)
    ├─> Analyzes context
    ├─> Determines: This is asking about account status, not trading ticker "ANY"
    ├─> Sets: request_type = "status_query"
    └─> Sets: ticker = "" (empty, no specific ticker mentioned)
    ↓
CLI Router
    ├─> Checks: request_type == "status_query"
    ├─> Checks keywords: "order" in input
    └─> Routes to: _handle_orders_request()
    ↓
Orders Handler
    └─> Shows open/pending orders

Trade Request Flow

User Input: "buy ANY at market"
    ↓
LLM Parser
    ├─> Analyzes context
    ├─> Determines: User wants to trade ticker symbol "ANY"
    ├─> Sets: request_type = "trade"
    └─> Sets: ticker = "ANY", action = "buy"
    ↓
CLI Router
    ├─> Checks: request_type == "trade"
    └─> Routes to: _handle_trade_request()
    ↓
Trade Orchestrator
    └─> Processes trade for ticker "ANY"

Implementation Details

1. TradeRequest Model Enhancement

File: src/core/models.py

Added request_type field:

@dataclass
class TradeRequest:
    ticker: str
    action: str  # "review", "buy", "sell"
    request_type: str = "trade"  # "trade" or "status_query" (LLM-determined)
    quantity: Optional[int] = None
    price: Optional[float] = None
    # ... other fields

2. LLM Parser Tool Schema

File: src/parsers/llm_parser.py

Enhanced tool definition with request_type:

tools = [
    {
        "name": "parse_trade_request",
        "parameters": {
            "properties": {
                "request_type": {
                    "type": "string",
                    "enum": ["trade", "status_query"],
                    "description": "Type of request: 'trade' for buy/sell/analyze ticker,
                                   'status_query' for asking about orders/positions/portfolio"
                },
                "ticker": {
                    "type": "string",
                    "description": "Stock ticker symbol. Empty string if status_query
                                   with no specific ticker."
                },
                # ... other fields
            }
        }
    }
]

3. Enhanced LLM Prompt

Key Addition: Context-aware classification guidance

First, determine the request_type:

- "trade" = User wants to buy, sell, or analyze a specific ticker
  Examples: "buy AAPL", "is SPY good?", "sell my TSLA"

- "status_query" = User is asking about their account status, orders, or positions
                   WITHOUT specifying intent to trade
  Examples: "any open orders?", "what positions do I have?", "show my portfolio"

  Note: Words like "any", "what", "show", "check" at the start usually indicate
        status queries, NOT ticker symbols!

4. Simplified CLI Routing

File: src/cli/cli_session.py

Routing Strategy:

Fast keyword routing for system features (scheduler, alerts)
LLM classification for everything else (trades, orders, positions, portfolio)

async def _process_request(self, user_input: str):
    input_lower = user_input.lower()

    # System features: Use keywords (not ambiguous)
    if any(word in input_lower for word in ["scheduler", "schedule", "routine"]):
        await self._handle_scheduler_request(user_input)

    elif any(word in input_lower for word in ["alert", "approaching"]) and "check" in input_lower:
        await self._handle_alerts_request(user_input)

    else:
        # Everything else: Let LLM classify
        await self._handle_trade_or_status_request(user_input)

LLM-Based Router:

async def _handle_trade_or_status_request(self, user_input: str):
    # Call LLM parser to classify
    request = await self.orchestrator.parser.parse(user_input, self.user_id)

    if request.request_type == "status_query":
        # Status query: route based on content keywords
        input_lower = user_input.lower()

        if any(word in input_lower for word in ["order", "orders"]):
            await self._handle_orders_request(user_input)
        elif any(word in input_lower for word in ["position", "positions"]):
            await self._handle_portfolio_request(user_input)
        else:
            await self._handle_portfolio_request(user_input)  # default

    else:
        # Trade request: process through orchestrator
        await self._handle_trade_request(user_input)

Benefits

1. Scalability

No special cases needed for ambiguous tickers:

✅ Handles: ANY, ALL, WHAT, WHO, WHEN, WHERE, WHY, HOW, etc.
✅ Future-proof: Works for any new ticker without code changes
✅ Natural: User can ask "what positions?" without triggering ticker "WHAT"

2. Context Understanding

LLM uses semantic meaning, not just keywords:

"any open orders?" → Asking about account status
"buy ANY" → Trading ticker symbol ANY
"what's the price of SPY?" → Asking about SPY ticker (trade context)
"what do I own?" → Asking about holdings (status query)

3. Maintainability

Reduced code complexity:

Before: 40+ hardcoded keyword patterns across 6 priority levels
After: 2 keyword checks (scheduler, alerts) + 1 LLM classification

4. Performance

Minimal overhead:

LLM call happens anyway for trade parsing
Now serves dual purpose: classification + extraction
Single parser call, no additional latency
Fast keyword routing still used for scheduler/alerts

Edge Cases Handled

Case 1: Ambiguous Words as Tickers

Input: "any open orders?"

LLM Analysis:

Context: Question about status ("any...?" pattern)
No action verb for trading (buy/sell/analyze)
Request type: status_query
Ticker: "" (empty)

Result: Routes to orders handler ✅

Input: "is ANY a good buy at market?"

LLM Analysis:

Context: Asking to analyze ticker for trading
Action: "review" (evaluate for purchase)
Specific ticker mentioned: ANY
Request type: trade
Ticker: "ANY"

Result: Routes to trade handler, analyzes ticker ANY ✅

Case 2: Natural Phrasings

Input: "what positions do I have?"

LLM Analysis:

Context: Asking about current holdings
No trading intent
Request type: status_query

Result: Shows portfolio ✅

Input: "what's AAPL trading at?"

LLM Analysis:

Context: Asking about ticker price (implies analysis)
Ticker: AAPL
Request type: trade (checking for potential entry)

Result: Analyzes AAPL ✅

Case 3: Compound Queries

Input: "show me my SPY position"

LLM Analysis:

Context: Status query about specific ticker
Request type: status_query
Ticker: "SPY" (specific position)

Result: Shows SPY position details with targets/stops ✅

Testing Guidelines

Test Coverage Required

Ambiguous ticker words:
- "any open orders?" → status
- "buy ANY" → trade ANY
- "what positions?" → status
- "is WHAT a good buy?" → trade WHAT
Natural language variations:
- "show orders" → status
- "check my holdings" → status
- "analyze SPY" → trade
- "how much buying power?" → status
Edge cases:
- "any positions in SPY?" → status (specific ticker position)
- "buy 10 shares" → trade (missing ticker, should error)
- "scheduler status" → scheduler (keyword bypass)

Manual Testing

Run through docs/features/05_interactive_cli_test_plan.md test cases and verify:

All status queries route correctly (no ticker misinterpretation)
All trade requests route correctly (even with ambiguous tickers)
No regression in scheduler/alerts routing
LLM classification logs show correct reasoning

Performance Considerations

LLM Call Overhead

Measurement:

LLM parser call: ~200-500ms (GPT-4o-mini)
Keyword routing: ~1ms

Optimization:

Scheduler/alerts still use fast keywords (0 LLM calls)
Trade/status queries: 1 LLM call (already needed for parsing)
No additional latency vs. old approach

Cost

Token Usage per Request:

Prompt: ~150 tokens
Response: ~50 tokens
Total: ~200 tokens = $0.0001 per request (negligible)

Migration Notes

From Keyword-Based to LLM-Based

What Changed:

Removed: 40+ hardcoded keyword patterns in CLI
Added: request_type field to TradeRequest model
Enhanced: LLM parser prompt with classification logic
Simplified: CLI routing to 2 keyword checks + 1 LLM call

Backward Compatibility:

✅ All existing commands still work
✅ No changes to trade orchestrator or handlers
✅ Only CLI routing logic changed

Testing Required:

Run all test cases from issue #334
Test ambiguous ticker names (ANY, ALL, WHAT, etc.)
Verify scheduler/alerts still use fast routing
Check LLM classification accuracy

Future Enhancements

Potential Improvements

Multi-intent Detection:
- "buy 10 AAPL and show my portfolio"
- LLM could return multiple intents
Confidence Scores:
- LLM returns confidence in classification
- Fallback to keyword if low confidence
Caching Common Queries:
- "show portfolio" → cached classification
- Reduces LLM calls for repetitive queries
Learning from Corrections:
- Track when LLM misclassifies
- Add to prompt as examples

References

Commits:

0c99ba3 - LLM-based routing implementation
2329373 - Previous keyword-based approach (deprecated)
f373b4f - Routing bug fixes
f6b25e9 - Toggle command with mode indicator

Design Decisions:

Use LLM for semantic understanding, not pattern matching
Keep fast keywords for unambiguous system features
Single source of truth: LLM parser determines intent
Scalable: No special cases for individual tickers

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LLM-Based Routing Architecture

Overview

Core Principle

Architecture

Request Classification Flow

Trade Request Flow

Implementation Details

1. TradeRequest Model Enhancement

2. LLM Parser Tool Schema

3. Enhanced LLM Prompt

4. Simplified CLI Routing

Benefits

1. Scalability

2. Context Understanding

3. Maintainability

4. Performance

Edge Cases Handled

Case 1: Ambiguous Words as Tickers

Case 2: Natural Phrasings

Case 3: Compound Queries

Testing Guidelines

Test Coverage Required

Manual Testing

Performance Considerations

LLM Call Overhead

Cost

Migration Notes

From Keyword-Based to LLM-Based

Future Enhancements

Potential Improvements

Related Documentation

References

FilesExpand file tree

06_llm_routing.md

Latest commit

History

06_llm_routing.md

File metadata and controls

LLM-Based Routing Architecture

Overview

Core Principle

Architecture

Request Classification Flow

Trade Request Flow

Implementation Details

1. TradeRequest Model Enhancement

2. LLM Parser Tool Schema

3. Enhanced LLM Prompt

4. Simplified CLI Routing

Benefits

1. Scalability

2. Context Understanding

3. Maintainability

4. Performance

Edge Cases Handled

Case 1: Ambiguous Words as Tickers

Case 2: Natural Phrasings

Case 3: Compound Queries

Testing Guidelines

Test Coverage Required

Manual Testing

Performance Considerations

LLM Call Overhead

Cost

Migration Notes

From Keyword-Based to LLM-Based

Future Enhancements

Potential Improvements

Related Documentation

References