MLflow + Claude Code Demo

Demo of "5 Tips to Get More Out of Your Claude Code with MLflow" (blog post).

Quick Start

# 1. Set required environment variables
export DATABRICKS_HOST=https://your-workspace.cloud.databricks.com
export DELTA_TABLE=my_catalog.default.claude_code_traces

# Optional:
# export DATABRICKS_PROFILE=my-profile       # Databricks CLI profile
# export DB_USER=you@company.com             # auto-detected if not set
# export MLFLOW_TRACING_SQL_WAREHOUSE_ID=abc123  # for LLM judge evaluations

# 2. Run setup (creates venv, authenticates, enables tracing)
./setup.sh

# 3. Run the interactive demo
./run_demo.sh

# Or run a specific tip (1-5)
./run_demo.sh 2

Environment Variables

Variable	Required	Description
DATABRICKS_HOST	Yes	Your Databricks workspace URL (e.g. https://my-workspace.cloud.databricks.com)
DELTA_TABLE	Yes (for setup.sh)	Unity Catalog table for trace archival (e.g. catalog.schema.table)
DATABRICKS_PROFILE	No	Databricks CLI profile name (auto-derived if not set)
DB_USER	No	Databricks user email (auto-detected from CLI auth)
MLFLOW_TRACING_SQL_WAREHOUSE_ID	No	SQL warehouse ID for LLM judge evaluations
MLFLOW_EXPERIMENT_NAME	No	Experiment name (used by Python scripts if not passed as arg)

The 5 Tips

#	Tip	What it does	Script
1	Monitor Usage Patterns	Trace every Claude Code session with mlflow autolog claude	scripts/01_setup_tracing.sh
2	Measure Performance	Evaluate traces with custom scorers + LLM judges	scripts/02_evaluate_traces.py
3	MLflow as a Tool	Give Claude Code access to MLflow via MCP server + CLI	scripts/03_mlflow_as_tool.sh
4	Automate with Skills	Install MLflow skills: npx skills add mlflow/skills	scripts/04_install_skills.sh
5	MLflow Assistant	Claude Code embedded in the MLflow UI	scripts/05_assistant_info.sh

Architecture

Part 1: Observability (Passive)          Part 2: Integration (Active)
┌──────────────────────────┐            ┌──────────────────────────┐
│ Tip 1: Tracing           │            │ Tip 3: MCP + CLI         │
│   mlflow autolog claude  │──traces──> │   Claude Code <-> MLflow │
│                          │            │                          │
│ Tip 2: Evaluation        │            │ Tip 4: Skills            │
│   Custom scorers +       │<─traces──  │   npx skills add         │
│   LLM judges             │            │   mlflow/skills          │
└──────────────────────────┘            │                          │
                                        │ Tip 5: Assistant         │
                                        │   Claude Code in MLflow  │
                                        │   UI                     │
                                        └──────────────────────────┘

Project Structure

mlflow-claude-code-demo/
├── setup.sh                              # One-command setup
├── eval.sh                               # Run evaluations
├── run_demo.sh                           # Interactive demo runner
├── pyproject.toml                        # Python dependencies
├── configs/
│   └── claude_settings_mcp.json.template # MCP server config template (Tip 3)
├── scripts/
│   ├── 01_setup_tracing.sh              # Tip 1: Enable tracing
│   ├── 01b_trace_with_sdk.py            # Tip 1: SDK variant
│   ├── 02_evaluate_traces.py            # Tip 2: Custom scorers
│   ├── 03_mlflow_as_tool.sh             # Tip 3: MCP + CLI setup
│   ├── 04_install_skills.sh             # Tip 4: Install skills
│   └── 05_assistant_info.sh             # Tip 5: Assistant overview
├── scorers/                              # Custom evaluation scorers
└── .claude/
    └── settings.json                     # Generated by setup.sh (not committed)

Prerequisites

• Python 3.10+
• MLflow >= 3.5.1
• Claude Code CLI
• uv (for MCP server)
• Databricks CLI (for authentication)

References

• Blog Post
• MLflow Claude Code Tracing Docs
• MLflow MCP Server Docs