Local LLM Instrumentation, Tracing, and Replay Platform

This project is a C++ terminal tool for observing local transformer inference. It can run a local GGUF model through llama.cpp, record runtime telemetry, replay saved runs, and show the data in a keyboard-driven terminal dashboard.

MVP Goal

Run a tiny local model safely, capture real llama.cpp runtime metrics, and inspect those metrics through replay or the terminal dashboard.

Features

• C++ CLI built with CMake.
• Local GGUF model execution through llama.cpp.
• Conservative CPU-only defaults for laptop-safe testing.
• Real runtime telemetry for load, prompt evaluation, generation evaluation, sampling, total time, token counts, and throughput.
• JSONL trace files that can be replayed after a run.
• Keyboard-driven terminal dashboard for real or fake telemetry.
• Callback-backed advanced tracing through llama.cpp cb_eval observations.
• Internal transformer submodule timing rows for selected layers, including attention, MLP, norm, and final norm rows.
• Real activation statistics for CPU-host tensors: mean, min, max, sparsity, sample count, NaN count, Inf count, and anomaly flags.
• Bounded real attention matrix capture for selected layer/head/window settings.
• Attention heatmap contrast controls and fullscreen inspection mode.
• Fixed-size tensor observation ring buffer so advanced capture memory stays bounded.
• Validation scripts for runtime telemetry, advanced schema, layer timings, activation stats, attention matrices, and CLI safety checks.

Requirements

• macOS or Linux
• C++20 compiler
• CMake 3.20+
• Git
• Homebrew on macOS

Quick Setup

From a fresh clone:

git clone https://github.com/parzival1821/llm_monitoring.git
cd llm_monitoring

Install the required local tools:

brew install cmake llama.cpp cpplint

Build the C++ app:

make build

Run the C++ linter:

make lint

Download the tiny local model:

make model

Run a safe smoke test:

make smoke

Generate and replay fake telemetry:

make telemetry-demo
make replay

Open the dashboard for the fake data:

make dashboard-fake

Ask a prompt and print only the model output:

make ask PROMPT="Explain gravity simply" TOKENS=64

Replay the real telemetry from that prompt:

make replay-real

Open the terminal dashboard for that real run:

make dashboard-real

If the answer stops mid-sentence, increase TOKENS:

make ask PROMPT="Explain gravity simply" TOKENS=128

Keep THREADS=2 while testing on a laptop.

Compile And Run For Testing

Use this sequence for a complete local verification pass:

make build
make lint
make model
make test-telemetry
make test-advanced-schema
make test-advanced-flags
make test-layer-timings
make test-activation-stats
make test-attention-matrix

Run a simple prompt and inspect the generated telemetry:

make ask PROMPT="Explain gravity simply" TOKENS=64 THREADS=2 CTX=256
make replay-real
make dashboard-real

Run the advanced capture path used for the final demo:

./build/trace run \
  --model models/smollm2-135m-instruct-q4_k_m.gguf \
  --prompt "What is attention in transformers?" \
  --tokens 8 \
  --threads 2 \
  --ctx-size 256 \
  --trace-out runs/latest-run.jsonl \
  --advanced-trace \
  --capture-attention \
  --capture-activations \
  --capture-layers 0 \
  --capture-heads 0 \
  --capture-token-window 8 \
  --activation-sample-limit 512 \
  --quiet

Then replay or open the dashboard:

make replay-real
make dashboard-real

Build the Project

cmake -S . -B build
cmake --build build

This creates:

./build/trace

Or use Make:

make build

Install llama.cpp

The safest local setup is to install the prebuilt Homebrew package:

brew install llama.cpp

Verify:

llama-completion --help

The source build helper is still available at scripts/bootstrap_llama_cpp.sh, but use Homebrew first because it avoids a heavy local compile.

Add a Model

Create a models/ folder and place a small GGUF model inside it.

Example path:

models/tiny-model.gguf

The models/ folder is ignored by Git because model files are large.

For a quick local smoke test, download the tiny SmolLM2 model:

./scripts/download_smollm2_model.sh

Or:

make model

This creates:

models/smollm2-135m-instruct-q4_k_m.gguf

Run a Prompt

Simplest smoke test:

make smoke

Custom prompt:

make run PROMPT="Explain gravity simply" TOKENS=32

Clean model-only output after setup:

make ask PROMPT="Explain gravity simply" TOKENS=64

make ask still records real runtime telemetry in:

runs/latest-run.jsonl

Replay it with:

make replay-real

Open it in the terminal dashboard:

make dashboard-real

Equivalent full command:

./build/trace run \
  --model models/smollm2-135m-instruct-q4_k_m.gguf \
  --prompt "Gravity is" \
  --threads 2 \
  --tokens 8 \
  --ctx-size 256 \
  --trace-out runs/latest-run.jsonl

By default, trace hides llama.cpp's verbose logs and shows the generated text. To debug the underlying runtime, add:

--show-logs

To hide the [trace] header too, add:

--quiet

If your llama-cli is somewhere else:

./build/trace run \
  --llama-bin /path/to/llama-completion \
  --model models/tiny-model.gguf \
  --prompt "Explain gravity in one sentence" \
  --threads 2 \
  --trace-out runs/latest-run.jsonl

To run without writing telemetry, add:

--no-trace

Real Telemetry

Real prompt runs now save a JSONL replay file based on llama.cpp's own performance report. The file contains one event per runtime metric:

• llama.load: model load time
• llama.prompt_eval: prompt processing time and prompt token throughput
• llama.generation_eval: generated-token evaluation time and generation throughput
• llama.sampling: sampler overhead
• llama.samplers: sampler chain time
• llama.total: total run time and total token throughput
• llama.unaccounted: time not covered by the other buckets

Safe real-data test:

make ask PROMPT="Gravity is" TOKENS=8 THREADS=2 CTX=256
make replay-real
make dashboard-real

For raw llama.cpp logs plus telemetry:

make run-logs PROMPT="Name two colors" TOKENS=6 THREADS=2 CTX=256
make replay-real

Run the fuller safe validation script:

make test-telemetry

That script runs multiple tiny CPU-only prompts, validates the JSONL structure, checks that token counts and timings are positive, checks total tokens against prompt plus generation tokens, compares total time against the phase breakdown, and verifies the parsed JSONL values against raw llama.cpp performance logs.

Default prompt runs still record coarse llama.cpp runtime phases. Opt-in advanced trace runs also record callback-backed internal submodule timing rows for selected transformer layers.

Advanced Trace Flags

Advanced tracing is opt-in because it can force llama.cpp to synchronize graph nodes for inspection. Keep the defaults small while developing on a laptop.

Safe defaults:

• CPU-only source-backed runner
• THREADS=2 in Makefile examples
• CTX=256 when --advanced-trace is used without --ctx-size
• --capture-layers 0
• --capture-heads 0
• --capture-token-window 32
• --activation-sample-limit 4096, which also caps the tensor observation ring buffer

Run bounded tensor discovery:

make discover-tensors

This writes:

runs/tensor-discovery.tsv

Inspect useful candidates:

sed -n '1,40p' runs/tensor-discovery.tsv
rg "kq_soft_max|Qcur|Kcur|Vcur|ffn_swiglu|ffn_out" runs/tensor-discovery.tsv

Equivalent explicit command:

./build/trace run \
  --model models/smollm2-135m-instruct-q4_k_m.gguf \
  --prompt "Gravity is" \
  --tokens 4 \
  --threads 2 \
  --ctx-size 256 \
  --advanced-trace \
  --discover-tensors \
  --capture-attention \
  --capture-activations \
  --capture-layers 0 \
  --capture-heads 0 \
  --capture-token-window 32 \
  --activation-sample-limit 4096 \
  --tensor-discovery-out runs/tensor-discovery.tsv

Advanced mode records tensor metadata such as execution order, phase, name, role, op, dtype, shape, bytes, view/buffer flags, and CPU/host hints. It also captures real attention matrices from callback-visible kq_soft_max tensors and real activation statistics from CPU-host tensors when those captures are enabled. Captured tensor observations are stored in a fixed-size FIFO ring buffer so long prompts stay bounded by --activation-sample-limit.

Validate advanced flag safety checks:

make test-advanced-flags

Real Layer and Submodule Timings

Advanced tracing maps llama.cpp callback observations into logical transformer submodules:

• layers.N.attention
• layers.N.mlp
• layers.N.norm
• final_norm

Each internal timing row includes:

• op_type=submodule_timing
• layer_index and submodule
• capture_source=llama.cpp.cb_eval
• start_ms, end_ms, latency_ms, duration_ms, and cumulative_ms
• percent_of_total, latency_rank, and latency_status
• activation diagnostics such as mean, min, max, sparsity, activation_sample_count, activation_nan_count, and activation_inf_count when activation capture succeeds

The internal durations are computed from adjacent callback boundaries inside the same logical submodule and decode step, with a first-to-last observation span as a same-decode fallback for otherwise-zero aggregates. That keeps the math consistent while avoiding false attribution across skipped layers or token steps.

Runtime phase ranks and internal submodule ranks are intentionally separate. The slowest internal row is labeled internal_bottleneck, not plain bottleneck, so the dashboard does not imply it is the global runtime bottleneck.

Run a real advanced capture:

make advanced-smoke
make replay-real TRACE_OUT=runs/advanced-smoke.jsonl

Validate the real timing rows and derived math:

make test-layer-timings

This checks that runtime phase telemetry still exists, layer 0 attention, MLP, and norm rows are present, replay prints them, and the slowest internal submodule is marked as the internal bottleneck.

Real Attention And Activation Capture

Run a small advanced capture that records real layer timing rows, activation statistics, and one real attention head matrix:

./build/trace run \
  --model models/smollm2-135m-instruct-q4_k_m.gguf \
  --prompt "What is attention in transformers?" \
  --tokens 8 \
  --threads 2 \
  --ctx-size 256 \
  --trace-out runs/latest-run.jsonl \
  --advanced-trace \
  --capture-attention \
  --capture-activations \
  --capture-layers 0 \
  --capture-heads 0 \
  --capture-token-window 8 \
  --activation-sample-limit 512 \
  --quiet

Then inspect it:

make replay-real
make dashboard-real

Validation commands:

make test-activation-stats
make test-attention-matrix

What is real:

• submodule_timing rows come from llama.cpp cb_eval callback observations.
• Activation stats are sampled from real CPU-host tensors.
• Attention heatmap is real when the dashboard title says [real] and the footer shows src: llama.cpp.cb_eval.

Fallback states:

• [placeholder] is fake demo data, used only when no attention capture exists.
• [not captured] means an attention event was requested but no matrix payload was available in that run.
• Runtime-only prompt runs still replay correctly but do not include internal activation or attention payloads.

Advanced JSONL Schema

The JSONL schema is backward compatible. Old runtime-only files without the advanced fields still replay. New advanced events may include:

• layer_index: transformer layer number, or -1 when not layer-specific
• submodule: logical area such as embedding, attention, mlp, norm, or output
• head_index: attention head number, or -1 when not head-specific
• capture_source: source of the data, such as llama.cpp.cb_eval
• capture_available: whether the requested capture was available
• capture_truncated: whether the capture was clipped by safety limits
• attention_tokens: token labels used by an attention capture
• attention_shape: matrix shape for attention data
• attention_matrix: bounded matrix values for replay/dashboard use
• activation_sample_count: number of activation values sampled
• activation_nan_count: number of sampled NaN values
• activation_inf_count: number of sampled infinite values
• anomaly_flags: labels such as max_gt_6 or nan_seen

Validate schema parsing and replay output:

make test-advanced-schema

This checks both:

• a legacy runtime-only fixture
• an advanced fixture with attention and activation fields

Returned Values

Each JSONL line is one telemetry event. The main fields are:

• id: event number in the run
• timestamp: when the event was recorded
• layer_name: runtime phase name, such as llama.prompt_eval
• layer_type: runtime for phase rows, or a submodule type such as attention
• op_type: operation name, such as prompt_eval, total, or submodule_timing
• shape: [token_count] for token-based runtime phases, or tensor shape for internal rows
• dtype: n/a for runtime phases, or the observed tensor dtype for internal rows
• device: currently CPU because the safe default disables GPU layers
• latency_ms: time for that runtime phase in milliseconds
• layer_index, submodule, head_index: optional advanced layer metadata
• capture_source, capture_available, capture_truncated: optional advanced capture metadata
• attention_tokens, attention_shape, attention_matrix: optional bounded attention payload; empty matrices are omitted from newly written JSONL
• activation_sample_count, activation_nan_count, activation_inf_count, anomaly_flags: optional activation diagnostics
• token_count: number of tokens or eval runs reported for that phase
• throughput_tokens_per_second: token speed for token-based phases
• start_ms: derived start point for this phase in the run timeline
• end_ms: derived end point for this phase in the run timeline
• duration_ms: derived duration, currently equal to latency_ms
• delta_from_previous_ms: derived elapsed time since the previous relevant timeline marker
• cumulative_ms: derived elapsed time up to this event
• percent_of_total: derived share of llama.cpp inference total; setup/load events use 0 because llama.cpp reports load time separately
• ms_per_token: derived average milliseconds per token when token count exists
• latency_rank: derived ranking by duration, where 1 is the slowest non-aggregate event
• timing_role: derived role such as setup, phase, detail, internal, or aggregate
• latency_status: derived label such as normal, high, bottleneck, internal_bottleneck, or aggregate
• note: extra context, currently model path and prompt character count

The numeric activation fields mean, min, max, and sparsity are present for future layer-level telemetry, but they are 0 for current runtime-only events.

Commands

./build/trace --help
./build/trace demo --out runs/fake-run.jsonl
./build/trace replay --file runs/fake-run.jsonl
./build/trace run --model <model.gguf> --prompt <text> [--threads 2] [--tokens 32] [--trace-out <run.jsonl>] [--no-trace] [--show-logs] [--quiet] [--advanced-trace] [--discover-tensors] [--capture-attention] [--capture-activations] [--capture-layers 0] [--capture-heads 0] [--capture-token-window 32] [--activation-sample-limit 4096]
./build/trace dashboard [--file <run.jsonl>]

demo writes fake transformer telemetry. run writes real runtime telemetry by default. replay reads JSONL telemetry and prints event summaries. dashboard opens the terminal UI; without --file it uses fake demo data, and with --file it opens a saved JSONL run.

Dashboard

The dashboard is a TUI, meaning a UI inside the terminal. It is not a browser app.

Full demo notes, panel explanations, screenshots, and limitations are in docs/demo.md.

Fake telemetry dashboard:

Real telemetry dashboard:

Advanced layer timing:

Activation stats:

Real attention heatmap:

Fake data demo:

make dashboard-fake

Real run demo:

make ask PROMPT="What is attention in transformers?" TOKENS=48 THREADS=2 CTX=256
make dashboard-real

Useful keys:

• j / k or arrow keys: move through events
• Tab / Shift+Tab: cycle focused panels
• h / l or left/right arrows: step replay mode
• Space: play/pause replay mode
• + / -: adjust attention heatmap contrast when the heatmap is focused
• r: reset attention heatmap contrast when the heatmap is focused
• F: toggle fullscreen heatmap mode when the heatmap is focused
• q or Esc: quit

Current Status

• C++ CLI and CMake build are in place.
• Local llama-completion runner uses conservative CPU defaults.
• Fake telemetry demo and JSONL replay flow are available.
• Real llama.cpp runtime telemetry is captured for prompt runs.
• Derived metrics include timeline points, percent of total, throughput, ms/token, rank, and status labels.
• Advanced trace emits real callback-backed layers.0.attention, layers.0.mlp, and layers.0.norm timing rows.
• Advanced trace can capture real activation stats and a bounded real attention matrix for selected layer/head/window settings.
• Keyboard-driven terminal dashboard is available for fake and real telemetry.

Assumptions And Verification Notes

• Model files are not committed. Run make model or place a GGUF model under models/.
• The default demo assumes the tiny SmolLM2 GGUF model and CPU execution.
• Keep THREADS=2, CTX=256, and small TOKENS values during laptop testing.
• Advanced capture is intentionally bounded by --capture-layers, --capture-heads, --capture-token-window, and --activation-sample-limit.
• The dashboard reads saved JSONL traces. It is not a live tail while inference is still running.
• Load time is reported separately from llama.cpp inference total, so load rows are treated as setup and do not contribute to %TOT.
• Internal submodule timings are diagnostic callback-boundary timings, not profiler-grade kernel timings.
• Attention heatmap data is real only when the panel title says [real] and the footer shows src: llama.cpp.cb_eval; otherwise the dashboard clearly marks placeholder or not-captured states.
• Verification-friendly extras include raw llama.cpp log comparison, replayable JSONL fixtures, screenshot artifacts, ring-buffer memory bounding, anomaly labels, heatmap contrast, and fullscreen heatmap mode.