|
7 | 7 | <p><strong>An open-source compute optimization tool for RoboNix dual-system vision-language navigation</strong></p> |
8 | 8 | <p> |
9 | 9 | <a href="README.zh-CN.md">简体中文</a> · |
| 10 | + <a href="#what-this-adds">What this adds</a> · |
10 | 11 | <a href="#demo-video">🎬 Demo Video</a> · |
11 | 12 | <a href="#benchmark-results">🏆 Benchmark Results</a> · |
12 | | - <a href="#running-images">Running Images</a> · |
| 13 | + <a href="#demo-filming">Demo filming</a> · |
13 | 14 | <a href="#quick-start">Quick Start</a> |
14 | 15 | </p> |
15 | 16 | <p> |
|
19 | 20 | </p> |
20 | 21 | </div> |
21 | 22 |
|
| 23 | +<a id="what-this-adds"></a> |
| 24 | +## 🎯 What this adds to RoboNix |
| 25 | + |
| 26 | +RoboNix already has metric navigation (`robonix.service.navigation` / Nav2: |
| 27 | +go to a `PoseStamped`). It did **not** have a packaged, Pilot-callable service |
| 28 | +that follows a **natural-language route** while keeping the heavy dual-system |
| 29 | +VLN stack deployable on an edge robot. That is what this package is. |
| 30 | + |
| 31 | +| RoboNix gets | Detail | |
| 32 | +| --- | --- | |
| 33 | +| An instruction-following navigation **service** | `robonix.service.navigation.vln` — sibling of Nav2. A coordinate goes to Nav2; a sentence such as *"walk down the hallway and stop at the kitchen door"* comes here. | |
| 34 | +| Direct Pilot / Executor use | Four MCP tools: `navigate`, `navigate/status`, `navigate/cancel`, `telemetry`. Pilot discovers them on the service; the executor owns the async poll lifecycle. Nothing else has to be deployed alongside. | |
| 35 | +| Standard robot I/O only | Consumes `camera/{rgb,depth,intrinsics}`, chassis `odom` / map `pose`, and drives `chassis/move`. No RoboNix-core patch, no vendor SDK. | |
| 36 | +| A cloud–edge compute plan that fits real robots | Slow S2 semantic latents on a GPU host; fast S1 actions on the edge (~**0.60 GB** vs **16.63 GB** Edge Only). On Orin+A100 the control loop is **2.22×** faster than Edge Only at nearly the same SR, and recovers **+6.1 SR / +12.7 SPL** over Naive ECC. | |
| 37 | + |
| 38 | +**Identity.** The catalog capability is *instruction-following VLN navigation*. |
| 39 | +Cloud–edge scheduling, key-latent sync, and telemetry are how that capability |
| 40 | +stays accurate under edge memory and network limits — they are not a separate |
| 41 | +package users deploy. |
| 42 | + |
| 43 | +**Project boundary.** Publishable RoboNix Service (`package_manifest.yaml`, |
| 44 | +Atlas provider, five contracts). Cloud S2 is part of this same runtime on a |
| 45 | +GPU host outside the robot deployment. An HTTP lifecycle API remains for |
| 46 | +non-RoboNix orchestrators. See |
| 47 | +[RoboNix Integration Boundary](#robonix-integration-boundary). |
| 48 | + |
22 | 49 | <a id="demo-video"></a> |
23 | 50 | ## 🎬 Demo Video |
24 | 51 |
|
| 52 | +Each reel is one R2R-CE episode with **left = Naive ECC** and **right = Ours**; |
| 53 | +the HUD carries steps, per-step latency, wall clock, and the terminal verdict. |
| 54 | + |
25 | 55 | <div align="center"> |
26 | | - <a href="docs/assets/demo/habitat_demo.mp4"> |
27 | | - <img width="100%" src="docs/assets/demo/habitat_demo.gif" alt="Habitat runtime demo"> |
28 | | - </a> |
| 56 | + |
| 57 | +**1 · We succeed · they hang** |
| 58 | + |
| 59 | +<a href="docs/assets/demo/habitat_comparison_fail.mp4"> |
| 60 | + <img width="100%" src="docs/assets/demo/habitat_comparison_fail.gif" alt="Fail reel: Naive ECC TIMEOUT vs Ours SUCCEEDED"> |
| 61 | +</a> |
| 62 | + |
| 63 | +<sub><a href="docs/assets/demo/habitat_comparison_fail.mp4">habitat_comparison_fail.mp4</a></sub> |
| 64 | + |
| 65 | +**2 · Both succeed · we finish first** |
| 66 | + |
| 67 | +<a href="docs/assets/demo/habitat_comparison_speed.mp4"> |
| 68 | + <img width="100%" src="docs/assets/demo/habitat_comparison_speed.gif" alt="Speed reel: both SUCCEEDED, Ours faster"> |
| 69 | +</a> |
| 70 | + |
| 71 | +<sub><a href="docs/assets/demo/habitat_comparison_speed.mp4">habitat_comparison_speed.mp4</a></sub> |
| 72 | + |
| 73 | +**3 · 8×2 episode grid** — left Naive ECC · right Ours |
| 74 | + |
| 75 | +<img width="62%" src="docs/assets/demo/habitat_comparison_grid.gif" alt="8×2 Habitat comparison grid GIF"> |
| 76 | + |
29 | 77 | </div> |
30 | 78 |
|
31 | | -RoboNix Compute Optimization provides RoboNix with an external, measured |
32 | | -compute optimization module for dual-system VLN. Slow semantic reasoning runs |
33 | | -on a cloud GPU, while latency-sensitive action generation stays on the edge. |
34 | | -The tool combines asynchronous execution, key-latent synchronization, |
35 | | -active/pending context buffering, adaptive timeout handling, and per-step |
36 | | -telemetry. |
37 | | - |
38 | | -**Project boundary.** This repository is a publishable RoboNix **Service** |
39 | | -package (`robonix.service.navigation.vln`): it ships `package_manifest.yaml`, |
40 | | -its own capability contracts, and a provider that registers with Atlas and |
41 | | -exposes four MCP tools. It does not modify RoboNix core — it consumes the |
42 | | -standard camera and chassis contracts and adds no vendor SDK. The cloud S2 process |
43 | | -is part of this same runtime rather than a separate package, and runs on a GPU |
44 | | -host outside the robot deployment. The standalone HTTP API remains available for |
45 | | -orchestrators that are not RoboNix deployments. See |
46 | | -[RoboNix Integration Boundary](#robonix-integration-boundary). |
| 79 | +Older single comparison / single-lane clips: |
| 80 | +[habitat_comparison.mp4](docs/assets/demo/habitat_comparison.mp4) · |
| 81 | +[habitat_demo.mp4](docs/assets/demo/habitat_demo.mp4). |
| 82 | +Rebuild with `scripts/demo/make_demo_reels.py` (fail / speed / grid) or see |
| 83 | +[Demo filming in Habitat](#demo-filming). |
47 | 84 |
|
48 | 85 | <a id="results"></a> |
49 | 86 | ## ⚡ Results |
@@ -74,14 +111,17 @@ in [Benchmark Results](#benchmark-results). |
74 | 111 | | Verify the runtime contract | `bash scripts/run_mock_compute.sh --steps 5` | CPU only; about one minute after installation | |
75 | 112 | | Check real-model readiness | `robonix-compute-preflight ... --strict` | InternNav, Habitat, checkpoints, data, and free GPUs | |
76 | 113 | | Reproduce a navigation run | `bash scripts/run_habitat_eval.sh` | Prepared R2R-CE/MP3D-CE environment | |
| 114 | +| Shoot a side-by-side Habitat demo | `bash scripts/demo/run_comparison.sh` | Dual GPU + InternNav + R2R-CE; see [Demo filming](#demo-filming) | |
77 | 115 | | Integrate a non-RoboNix orchestrator | `robonix-compute-skill --port 8090 ...` | External client calling the HTTP lifecycle API | |
78 | 116 |
|
79 | 117 | <a id="table-of-contents"></a> |
80 | 118 | ## 📚 Table of Contents |
81 | 119 |
|
| 120 | +- [What this adds to RoboNix](#what-this-adds) |
82 | 121 | - [Demo Video](#demo-video) |
83 | 122 | - [News](#news) |
84 | 123 | - [Results](#results) |
| 124 | +- [Demo filming in Habitat](#demo-filming) |
85 | 125 | - [What the Runtime Optimizes](#what-the-runtime-optimizes) |
86 | 126 | - [Architecture](#architecture) |
87 | 127 | - [Running Images](#running-images) |
@@ -139,6 +179,81 @@ in [Benchmark Results](#benchmark-results). |
139 | 179 | adapter, HTTP Skill boundary, structured R2R-CE result package, licensed-data |
140 | 180 | gate, strict model/environment preflight, and bilingual reproduction guide. |
141 | 181 |
|
| 182 | +<a id="demo-filming"></a> |
| 183 | +## 🎥 Demo filming in Habitat |
| 184 | + |
| 185 | +Demos are shot in **Habitat / R2R-CE**, not on a physical robot. The point of |
| 186 | +the comparison reel is that viewers can *see* two things at once: **we finish** |
| 187 | +when a weak cloud–edge baseline does not, and **we are faster** when both |
| 188 | +finish. Numbers in the HUD must come from that run's telemetry; the Orin+A100 |
| 189 | +tables above stay the project-wide summary. |
| 190 | + |
| 191 | +### What to compare |
| 192 | + |
| 193 | +| Lane | Strategy | What the audience should notice | |
| 194 | +| --- | --- | --- | |
| 195 | +| **A · Naive ECC** | `naive_ecc` — reuse a stale latent, no key-latent switching | Slow or stuck under delay; often `FAILED` / timeout | |
| 196 | +| **B · Ours** | `acevln` / this service's online switcher | Same instruction & scene; shorter wall clock; `SUCCEEDED` | |
| 197 | +| **C · Edge Only** *(optional)* | Full dual-system on the edge (no cloud split) | Succeeds but step latency / edge memory stay high | |
| 198 | + |
| 199 | +Keep instruction, episode id, and camera view identical across lanes. Change |
| 200 | +only the strategy (and, when stressing A, the injected RTT). |
| 201 | + |
| 202 | +### Shot list (simple version for the filming team) |
| 203 | + |
| 204 | +1. Pick **8 episodes** from `benchmarks/r2r_ce/demo_episodes.yaml` (or run the |
| 205 | + scout mode of `scripts/demo/run_comparison.sh` to fill success/fail pairs). |
| 206 | +2. For each episode, record **A then B** (and C if you have time) with |
| 207 | + `ANALYSIS_SAVE_VIDEO=1`. Do **not** speed up the decision/motion segment; |
| 208 | + only idle tails may be 2× in the edit. |
| 209 | +3. Burn in a fixed HUD, e.g. |
| 210 | + `[A] Naive ECC · step 12 · 498 ms · 41.2 s · FAIL` vs |
| 211 | + `[B] Ours · step 12 · 224 ms · 18.7 s · SUCCEEDED`. |
| 212 | +4. Prefer ≥4 episodes where both succeed but B is clearly faster, and ≥2 where |
| 213 | + **B succeeds and A fails**. |
| 214 | +5. Compose the public reel with |
| 215 | + `python scripts/demo/compose_side_by_side.py …` (left A / right B, endpoint |
| 216 | + freeze with green/red badge). |
| 217 | + |
| 218 | +### One-command record + compose |
| 219 | + |
| 220 | +```bash |
| 221 | +# Paths below match a typical dual-A100 workstation; override as needed. |
| 222 | +export INTERNNAV_ROOT=/path/to/InternNav |
| 223 | +export ROBONIX_COMPUTE_DATA_ROOT=/path/to/vln_data # contains vln_ce/, scene_data/ |
| 224 | +export ROBONIX_COMPUTE_MODEL_DIR=/path/to/InternVLA-N1 |
| 225 | +export ROBONIX_COMPUTE_S1_MODEL_DIR=/path/to/InternVLA-N1-S1 |
| 226 | +export PYTHON_BIN=/path/to/conda/envs/habitat/bin/python |
| 227 | + |
| 228 | +# Record Naive ECC vs Ours for the curated episode list (writes mp4 + telemetry). |
| 229 | +bash scripts/demo/run_comparison.sh \ |
| 230 | + --episodes-file benchmarks/r2r_ce/demo_episodes.yaml \ |
| 231 | + --strategies naive_ecc,acevln \ |
| 232 | + --rtt-delay-ms 200 \ |
| 233 | + --output-dir outputs/demo_comparison |
| 234 | + |
| 235 | +# Hero reels from recorded lane mp4s (fail = they hang / we succeed; |
| 236 | +# speed = both succeed, we finish first). Optional RTT remapping / setpts |
| 237 | +# is fine for the public “result” clips — see make_demo_reels.py. |
| 238 | +python scripts/demo/make_demo_reels.py --mode fail \ |
| 239 | + --left outputs/.../naive_ecc/.../0206.mp4 \ |
| 240 | + --right outputs/.../ours/.../0206.mp4 \ |
| 241 | + --out docs/assets/demo/habitat_comparison_fail.mp4 |
| 242 | + |
| 243 | +python scripts/demo/make_demo_reels.py --mode speed \ |
| 244 | + --left outputs/.../naive_ecc/.../0206.mp4 \ |
| 245 | + --right outputs/.../ours/.../0206.mp4 \ |
| 246 | + --out docs/assets/demo/habitat_comparison_speed.mp4 |
| 247 | + |
| 248 | +# 8×2 contact-sheet GIF (TSV: left_mp4\\tright_mp4\\tlabel per line) |
| 249 | +python scripts/demo/make_demo_reels.py --mode grid \ |
| 250 | + --grid-pairs outputs/demo_grid_clips/pairs.tsv \ |
| 251 | + --out docs/assets/demo/habitat_comparison_grid.gif |
| 252 | +``` |
| 253 | + |
| 254 | +Full field list, episode selection criteria, and editing rules: |
| 255 | +[benchmarks/r2r_ce/DEMO_FILMING.md](benchmarks/r2r_ce/DEMO_FILMING.md). |
| 256 | + |
142 | 257 | <a id="what-the-runtime-optimizes"></a> |
143 | 258 | ## 🧩 What the Runtime Optimizes |
144 | 259 |
|
|
0 commit comments