License notice This project is source-available under the PolyForm Noncommercial License 1.0.0 — free for personal, research, teaching, and internal-evaluation use.

Any commercial use, redistribution as part of a commercial product, or paid hosted deployment requires a separate commercial license. See COMMERCIAL.md or contact https://github.com/AnubhabBanerjee.

Unauthorized commercial use or redistribution is a violation of the license terms.

---

---

VLA-Edge-Backend

A deadline-aware, memory-bounded inference runtime for streaming Vision-Language(-Action) models on edge GPUs.

Built by Anubhab Banerjee — LinkedIn · GitHub

Skills demonstrated: CUDA kernel authorship, GPU memory management, real-time systems engineering, lock-free concurrency, C++17, robotics / VLA inference.

---

Why this exists

Physical AI has a systems problem: intelligence must operate continuously under hard memory and latency constraints. Existing LLM inference runtimes fail in robotics due to:

• Memory Exhaustion: Continuous streaming of high-dimensional vision tokens overflows the strict VRAM limits of edge devices.
• Latency Spikes & Deadline Misses: Standard runtimes compute without time bounds. VLA models stalling during reasoning miss physical deadlines (e.g., 30Hz), leading to hardware crashes.
• Frequency Mismatch: High-speed camera ingest (60Hz) clashes with slower VLM reasoning (10Hz).

Generic LLM inference runtimes are built for chatbots, not robots. Point a VLA model at a live camera feed and three things break at once:

• VRAM explodes. Streaming vision tokens continuously overflow a fixed edge-GPU memory budget — there's no such thing as "just add more context."
• Deadlines get missed silently. Standard runtimes have no concept of a physical control deadline (e.g. 30Hz). A slow reasoning step doesn't just feel laggy — it can crash a physical system.
• Frequencies don't match. Cameras run at 60Hz. VLA reasoning runs at ~10Hz. Naive pipelines either block perception on reasoning or silently pile up stale frames.

This repo builds a runtime that treats these as first-class constraints instead of ignoring them: a hard memory budget with semantic (not FIFO) eviction, a lock-free async perception pipeline, and a deadline-aware admission controller that refuses work it can't finish in time rather than missing a deadline.

---

Architecture

flowchart LR
    subgraph Perception["Async Perception Pipeline (60Hz)"]
        CAM[Camera / dataset replay] --> BUF[Lock-free double buffer]
    end

    subgraph Runtime["Deadline-Aware Runtime (33ms cycle)"]
        BUF -->|freshest frame only| ADM{Admission Controller}
        ADM -->|budget available| VLM[Hand-written CUDA\nTransformer + Vision Encoder]
        ADM -->|budget exceeded| FB[Fallback action\nrepeat last command]
        VLM <--> KV[(KV Manager\nsemantic eviction)]
    end

    VLM --> ACT[Action]
    FB --> ACT

Loading

Component	Responsibility
KV Manager	Enforces a hard token budget; evicts the most redundant (cosine-similarity-based), not the oldest, frame when over budget
Async Perception Pipeline	Decouples 60Hz camera ingest from ~10Hz reasoning via a lock-free double buffer — reasoning always sees the freshest frame, never a stale backlog
Deadline-Aware Admission Controller	Estimates a reasoning chunk's cost before launching it; refuses admission (and falls back) rather than blow a 33ms physical deadline

---

Evaluation

The runtime is benchmarked against FIFO and sliding-window KV-cache baselines on four headline metrics: deadline miss rate, p50/p95/p99 latency, fallback/degraded-action rate, and peak VRAM against the 8GB ceiling — plus a saliency-retention check for the semantic eviction policy specifically.

One number is worth stating here rather than saving for that write-up: steady-state, a single admitted reasoning chunk (32-token prefill + 8-token decode) currently costs roughly 3.3 seconds on this Hopper GPU — not the 33ms DEADLINE_MS the admission controller is built to enforce. The scaffolding for the deadline is real and fully instrumented; the transformer engine's raw throughput closing the gap to actually live inside it is not, yet.

Measured results, full methodology, and honest caveats (untrained vision encoder, synthetic timebase, etc.) will be published as a standalone write-up rather than duplicated here.

---

Tech stack

• Bare-Metal Runtime: C++17 / CUDA 12.4, featuring hand-written kernels (no cuBLAS, no libtorch, no ONNX), CMake build, targeting NVIDIA Hopper (sm_90)
• Model: Qwen2.5-Coder-1.5B-Instruct decode-only transformer, implemented from scratch (RMSNorm, RoPE, GQA attention, SwiGLU) — no inference framework dependency, validated against a HuggingFace reference forward pass (cosine similarity ≥ 0.999)
• Tooling only (not in the runtime hot path): Python 3.10 for dataset preparation (Open X-Embodiment), benchmarking orchestration, and plotting

---

Repository structure

physical-ai/
├── CMakeLists.txt              # Build: physical_ai_runtime + physical_ai_tests
├── README.md
├── overview_image.png
│
├── include/physical_ai/        # Public C++ headers
│   ├── common.hpp              # Compile-time constants, CUDA helpers, VRAM tracker
│   ├── kernels.hpp             # Kernel launch wrappers (RMSNorm, RoPE, GQA, SwiGLU, matmul)
│   ├── transformer_engine.hpp  # Layer forward, embed, logits, scratch buffers
│   ├── vision_encoder.hpp      # Deterministic patch embedder (32 tokens × 1536)
│   ├── safetensors_loader.hpp  # Qwen2.5 weight upload from local safetensors
│   ├── kv_manager.hpp          # KV cache API (semantic / fifo / sliding_window)
│   ├── perception_pipeline.hpp # Lock-free double-buffered frame ingest
│   ├── admission_controller.hpp# Deadline-aware admit/refuse + fallback
│   └── metrics.hpp             # Headline metrics + JSON export
│
├── src/
│   ├── main_runtime.cpp        # Benchmark CLI entry point
│   ├── common.cu               # Instrumented cudaMalloc/cudaFree, timing helpers
│   ├── transformer_engine.cu   # Hand-written transformer forward pass
│   ├── vision_encoder.cu
│   ├── safetensors_loader.cpp
│   ├── kv_manager_semantic.cu  # Cosine-similarity eviction policy
│   ├── kv_manager_fifo.cu      # FIFO baseline
│   ├── kv_manager_sliding_window.cu
│   ├── perception_pipeline.cpp # 60 Hz dataset replay producer
│   ├── admission_controller.cpp
│   ├── metrics.cpp
│   └── kernels/                # CUDA kernels (no cuBLAS)
│       ├── rmsnorm.cu
│       ├── rope.cu
│       ├── attention_gqa.cu
│       ├── swiglu_mlp.cu
│       ├── matmul.cu
│       └── vision_patch_embed.cu
│
├── tests/                      # Custom test runner (no gtest)
│   ├── test_main.cpp
│   ├── test_runner.hpp
│   ├── test_rmsnorm.cpp
│   ├── test_rope.cpp
│   ├── test_swiglu_mlp.cpp
│   ├── test_attention_gqa.cpp
│   ├── test_vision_patch_embed.cpp
│   ├── test_kv_manager_semantic.cpp
│   ├── test_kv_manager_fifo.cpp
│   ├── test_kv_manager_sliding_window.cpp
│   ├── test_admission_controller.cpp
│   ├── test_transformer_forward.cpp   # HF-reference correctness gate
│   └── reference_dump/         # Frozen HF forward-pass tensors for the gate
│
├── tools/                      # Python — offline only, not in runtime hot path
│   ├── prepare_dataset.py      # OpenX → raw_clip.bin + critical-event meta
│   ├── inspect_safetensors.py  # Phase-1 tensor inventory
│   ├── inspect_openx_schema.py # Phase-1 dataset schema evidence
│   ├── dump_hf_reference.py    # Generate tests/reference_dump/ from HF
│   ├── run_benchmarks.py       # Orchestrate 3-policy benchmark sweep
│   ├── plot_results.py         # Headline metric plots from JSON
│   ├── generate_report.py      # Draft write-up from benchmark JSON
│   ├── safetensors_tensor_list.txt
│   └── openx_schema_dump.txt
│
└── third_party/
    └── json.hpp                # nlohmann/json (metrics JSON output)

Not shipped in the repo (generated locally at build/benchmark time):

Path	Created by
build/	cmake --build
data/raw_clip.bin, data/raw_clip_meta.json, data/system_prompt_tokens.bin	tools/prepare_dataset.py
results/*.json, results/*.png	physical_ai_runtime, tools/plot_results.py

---

Getting started

# Build
cmake -S . -B build && cmake --build build -j

# Run the test suite
./build/physical_ai_tests

# Run a benchmark
./build/physical_ai_runtime --policy semantic --duration_sec 120 --output results/semantic_smoke.json