Agentic Intent Classifier

agentic-intent-classifier is a multi-head query classification stack for conversational traffic.

Quickstart (recommended): run from Hugging Face Hub

This is the easiest way for developers to test the full production stack (multitask intent + IAB + calibration) without training locally.

from transformers import pipeline

clf = pipeline(
    "admesh-intent",
    model="admesh/agentic-intent-classifier",
    trust_remote_code=True,
)

out = clf("Which laptop should I buy for college?")
print(out["model_output"]["classification"]["intent"])
print(out["model_output"]["classification"]["iab_content"])
print(out["meta"])

If you’re running in Colab/Kaggle and see Torch version conflicts, follow COLAB_SETUP.md.

Latency / inference timing (developer quick check)

The first call includes model/code loading; measure latency after a warm-up call.

Single query:

import time
from transformers import pipeline

clf = pipeline("admesh-intent", model="admesh/agentic-intent-classifier", trust_remote_code=True)
q = "Which laptop should I buy for college?"

_ = clf("warm up")
t0 = time.perf_counter()
out = clf(q)
dt_ms = (time.perf_counter() - t0) * 1000

print(f"latency_ms={dt_ms:.1f}")
print(out["model_output"]["classification"]["intent"])

Warm p50 / p95 over 20 runs:

import time, statistics

times = []
for _ in range(20):
    t0 = time.perf_counter()
    _ = clf(q)
    times.append((time.perf_counter() - t0) * 1000)

times_sorted = sorted(times)
print(f"p50={statistics.median(times):.1f}ms p95={times_sorted[int(0.95*len(times))-1]:.1f}ms mean={statistics.mean(times):.1f}ms")

It currently produces:

• intent.type
• intent.subtype
• intent.decision_phase
• iab_content
• calibrated confidence per head
• combined fallback / policy / opportunity decisions

The repo is beyond the original v0.1 baseline. It now includes:

• shared config and label ownership
• reusable model runtime
• calibrated confidence and threshold gating
• combined inference with fallback/policy logic
• request/response validation in the demo API
• repeatable evaluation and regression suites
• full-TSV IAB taxonomy retrieval support through tier4
• a local embedding index for taxonomy-node retrieval over IAB content paths
• a separate synthetic full-intent-taxonomy augmentation dataset for non-IAB heads
• a dedicated intent-type difficulty dataset and held-out benchmark with easy, medium, and hard cases
• a dedicated decision-phase difficulty dataset and held-out benchmark with easy, medium, and hard cases

Generated model weights are intentionally not committed.

Current Taxonomy

intent.type

• informational
• exploratory
• commercial
• transactional
• support
• personal_reflection
• creative_generation
• chit_chat
• ambiguous
• prohibited

intent.decision_phase

• awareness
• research
• consideration
• decision
• action
• post_purchase
• support

intent.subtype

• education
• product_discovery
• comparison
• evaluation
• deal_seeking
• provider_selection
• signup
• purchase
• booking
• download
• contact_sales
• task_execution
• onboarding_setup
• troubleshooting
• account_help
• billing_help
• follow_up
• emotional_reflection

iab_content

• candidates are derived from every row in data/iab-content/Content Taxonomy 3.0.tsv
• retrieval output supports tier1, tier2, tier3, and optional tier4

What The System Does

• runs three classifier heads:
  • intent_type
  • intent_subtype
  • decision_phase
• resolves iab_content through a local embedding index over taxonomy nodes plus generic label/path reranking
• applies calibration artifacts when present
• computes commercial_score
• applies fallback when confidence is too weak or policy-safe blocking is required
• emits a schema-validated combined envelope

What The System Does Not Do

• it is not a multi-turn memory system
• it is not a production-optimized low-latency serving path
• it is not yet trained on large real-traffic human-labeled intent data
• combined decision logic is still heuristic, even though it is materially stronger than the original baseline

Project Layout

• config.py: labels, thresholds, artifact paths, model paths
• model_runtime.py: shared calibrated inference runtime
• combined_inference.py: composed system response
• inference_intent_type.py: direct intent_type inference entrypoint
• inference_iab_classifier.py: direct supervised iab_content inference entrypoint
• schemas.py: request/response validation
• demo_api.py: local validated API
• iab_taxonomy.py: full taxonomy parser/index
• iab_classifier.py: supervised IAB runtime with taxonomy-aware parent fallback
• iab_retrieval.py: optional shadow retrieval baseline
• training/build_full_intent_taxonomy_dataset.py: separate synthetic intent augmentation dataset
• training/build_intent_type_difficulty_dataset.py: extra intent_type augmentation plus held-out difficulty benchmark
• training/build_decision_phase_difficulty_dataset.py: extra decision_phase augmentation plus held-out difficulty benchmark
• training/build_subtype_difficulty_dataset.py: extra intent_subtype augmentation plus held-out difficulty benchmark
• training/build_subtype_dataset.py: subtype dataset generation from existing corpora
• training/train_iab.py: train the supervised IAB classifier head
• training/build_iab_taxonomy_embeddings.py: build local IAB node embedding artifacts
• training/run_full_training_pipeline.py: full multi-head training/calibration/eval pipeline
• evaluation/run_evaluation.py: repeatable benchmark runner
• evaluation/run_regression_suite.py: known-failure regression runner
• evaluation/run_iab_mapping_suite.py: IAB behavior-lock regression runner
• evaluation/run_iab_quality_suite.py: curated IAB quality-target runner
• known_limitations.md: current gaps and caveats

Quickstart: Run From Hugging Face

Download the trained bundle and run inference in three lines — no local training required.

import sys
from huggingface_hub import snapshot_download

# Download the full bundle (models + calibration + code)
local_dir = snapshot_download(
    repo_id="admesh/agentic-intent-classifier",
    repo_type="model",
)
sys.path.insert(0, local_dir)

# Import and instantiate
from pipeline import AdmeshIntentPipeline
clf = AdmeshIntentPipeline()

# Classify
import json
result = clf("Which laptop should I buy for college?")
print(json.dumps(result, indent=2))

Or use the one-liner factory method:

from pipeline import AdmeshIntentPipeline   # after sys.path.insert above

clf = AdmeshIntentPipeline.from_pretrained("admesh/agentic-intent-classifier")
result = clf("I need a CRM for a 5-person startup")

Batch mode and custom thresholds are also supported:

# Batch
results = clf([
    "Best running shoes under $100",
    "How does gradient descent work?",
    "Buy noise-cancelling headphones",
])

# Custom confidence thresholds
result = clf(
    "Buy noise-cancelling headphones",
    threshold_overrides={"intent_type": 0.6, "intent_subtype": 0.35},
)

Verify artifacts and run a smoke test from the CLI:

cd "<local_dir>"
python3 training/pipeline_verify.py
python3 combined_inference.py "Which CRM should I buy for a 3-person startup?"

Pin a specific revision for reproducibility:

local_dir = snapshot_download(
    repo_id="admesh/agentic-intent-classifier",
    repo_type="model",
    revision="0584798f8efee6beccd778b0afa06782ab5add60",
)

---

Setup (for local training)

python3 -m venv .venv
source .venv/bin/activate
pip install -r agentic-intent-classifier/requirements.txt

Inference (local training path)

Run one query locally:

cd agentic-intent-classifier
python3 training/train_iab.py
python3 training/calibrate_confidence.py --head iab_content
python3 combined_inference.py "Which CRM should I buy for a 3-person startup?"

Run only the intent_type head:

cd agentic-intent-classifier
python3 inference_intent_type.py "best shoes under 100"

Run the demo API:

cd agentic-intent-classifier
python3 demo_api.py

Example request:

curl -sS -X POST http://127.0.0.1:8008/classify \
  -H 'Content-Type: application/json' \
  -d '{"text":"I cannot log into my account"}'

Infra endpoints:

curl -sS http://127.0.0.1:8008/health
curl -sS http://127.0.0.1:8008/version

Train only the IAB classifier head:

cd agentic-intent-classifier
python3 training/train_iab.py
python3 training/calibrate_confidence.py --head iab_content

The online iab_content path now uses the compact supervised classifier. Retrieval is still available as an optional shadow baseline.

Build the optional retrieval shadow index:

cd agentic-intent-classifier
python3 training/build_iab_taxonomy_embeddings.py

By default the shadow retrieval path uses Alibaba-NLP/gte-Qwen2-1.5B-instruct. The retrieval runtime applies the model's query-side instruction format and last-token pooling, matching the Hugging Face usage guidance. If you want to point retrieval at a different embedding model, set IAB_RETRIEVAL_MODEL_NAME_OVERRIDE before building the index.

Open-source users can swap in their own embedding model, but the contract is:

• query embeddings and taxonomy-node embeddings must be produced by the same model and model revision
• after changing models, you must rebuild artifacts/iab/taxonomy_embeddings.pt
• the repository only tests and supports the default model path out of the box
• not every Hugging Face embedding model is drop-in compatible with this runtime; some require custom pooling, query instructions, or trust_remote_code

Example override:

cd agentic-intent-classifier
export IAB_RETRIEVAL_MODEL_NAME_OVERRIDE=mixedbread-ai/mxbai-embed-large-v1
python3 training/build_iab_taxonomy_embeddings.py

This writes:

• artifacts/iab/taxonomy_nodes.json
• artifacts/iab/taxonomy_embeddings.pt

Training

Full local pipeline

cd agentic-intent-classifier
python3 training/run_full_training_pipeline.py

This pipeline now does:

1. build separate full-intent-taxonomy augmentation data
2. build separate intent_type difficulty augmentation + benchmark
3. train intent_type
4. build subtype corpus
5. build separate intent_subtype difficulty augmentation + benchmark
6. train intent_subtype
7. build separate decision_phase difficulty augmentation + benchmark
8. train decision_phase
9. train iab_content
10. calibrate all classifier heads, including iab_content
11. run regression/evaluation unless --skip-full-eval is used

Build datasets individually

Separate full-intent augmentation:

cd agentic-intent-classifier
python3 training/build_full_intent_taxonomy_dataset.py

Intent-type difficulty augmentation and benchmark:

cd agentic-intent-classifier
python3 training/build_intent_type_difficulty_dataset.py

Decision-phase difficulty augmentation and benchmark:

cd agentic-intent-classifier
python3 training/build_decision_phase_difficulty_dataset.py

Subtype difficulty augmentation and benchmark:

cd agentic-intent-classifier
python3 training/build_subtype_difficulty_dataset.py

Subtype dataset:

cd agentic-intent-classifier
python3 training/build_subtype_dataset.py

IAB embedding index:

cd agentic-intent-classifier
python3 training/build_iab_taxonomy_embeddings.py

Train heads individually

cd agentic-intent-classifier
python3 training/train.py
python3 training/train_subtype.py
python3 training/train_decision_phase.py

Calibration

cd agentic-intent-classifier
python3 training/calibrate_confidence.py --head intent_type
python3 training/calibrate_confidence.py --head intent_subtype
python3 training/calibrate_confidence.py --head decision_phase

Evaluation

Full evaluation:

cd agentic-intent-classifier
python3 evaluation/run_evaluation.py

Known-failure regression:

cd agentic-intent-classifier
python3 evaluation/run_regression_suite.py

IAB behavior-lock regression:

cd agentic-intent-classifier
python3 evaluation/run_iab_mapping_suite.py

IAB quality-target evaluation:

cd agentic-intent-classifier
python3 evaluation/run_iab_quality_suite.py

Threshold sweeps:

cd agentic-intent-classifier
python3 evaluation/sweep_intent_threshold.py

Artifacts are written to:

• artifacts/calibration/
• artifacts/evaluation/latest/

Google Colab

Use Colab for the full retraining pass if local memory is limited.

Clone once:

%cd /content
!git clone https://github.com/GouniManikumar12/agentic-intent-classifier.git
%cd /content/agentic-intent-classifier

If the repo is already cloned and you want the latest code, pull manually:

!git pull origin main

Full pipeline:

!python training/run_full_training_pipeline.py

If full evaluation is too heavy for the current Colab runtime:

!python training/run_full_training_pipeline.py \
  --iab-embedding-batch-size 32 \
  --skip-full-eval

Then run eval separately after training:

!python evaluation/run_regression_suite.py
!python evaluation/run_iab_mapping_suite.py
!python evaluation/run_iab_quality_suite.py
!python evaluation/run_evaluation.py

Current Saved Metrics

Generate fresh metrics with:

cd agentic-intent-classifier
python3 evaluation/run_evaluation.py

Do not treat any checked-in summary as canonical unless it was regenerated after the current code and artifacts were built. The IAB path is now retrieval-based, so older saved reports from the deleted hierarchy stack are not meaningful.

Latency Note

combined_inference.py is a debugging/offline path, not a production latency path.

Current production truth:

• per-request CLI execution is not a sub-50ms architecture
• production serving should use a long-lived API process with preloaded models
• if sub-50ms becomes a hard requirement, the serving path will need:
  • persistent loaded models
  • runtime optimization
  • likely fewer model passes or a shared multi-head model

Current Status

Current repo status:

• full 10-class intent.type taxonomy is wired
• subtype and phase heads are present
• difficulty benchmarks are wired for intent_type, intent_subtype, and decision_phase
• full-TSV IAB taxonomy retrieval is wired through tier4
• separate full-intent augmentation dataset is in place
• evaluation/runtime memory handling is improved for large IAB splits

The main remaining gap is not basic infrastructure anymore. It is improving real-world robustness, especially for:

• decision_phase
• intent_subtype
• confidence quality on borderline commercial queries
• real-traffic supervision beyond synthetic data