feat(python): add Cosmos Transfer pipeline, docs, and smoke test (3/4)

[youtalk]

Description

This is PR 3 of a 4-PR stack that ports NVIDIA Cosmos Transfer support from ue4-dev to ue5-dev. It is stacked on top of PR 2 (#9797) and should be reviewed/merged bottom-up (PR1 #9796 → PR2 #9797 → this PR → PR4).

Stacked PR — review bottom-up. Because a cross-fork pull request can only target an upstream branch, this PR is opened against ue5-dev and its diff therefore also contains the commits from PR 1 (#9796) and PR 2 (#9797). Please review only the commits unique to this branch and merge the stack bottom-up. Once #9796 and #9797 land in ue5-dev, this branch will be rebased and the diff will reduce to its own commits. The commits unique to this PR:

• feat(python): add Cosmos Transfer client/server example pipeline
• docs(cosmos): add Cosmos Transfer guide and smoke test
• fix(cosmos): route example_data mp4/log through Git LFS
• docs(changelog): note Cosmos Transfer pipeline and docs
• fix(cosmos): regenerate example logs/artifacts for ue5-dev (0.10.0)

PR3 adds the branch-independent Python client/server pipeline, documentation, and a smoke test that drive the C++ HD-map exporters introduced in PR2. None of this code depends on engine internals beyond the Python API already shipped by the lower PRs, so it can be reviewed largely on its own once the stack is in place.

What this PR adds:

• Cosmos Transfer client pipeline under PythonAPI/examples/nvidia/cosmos/client/. carla_cosmos_gen.py replays an InvertedAI log inside CARLA, captures the AOV (arbitrary-output-variable) frames — RGB, semantic segmentation, instance segmentation, depth, edges — and calls the six World.export_cosmos_* RPCs from PR2 (export_cosmos_crosswalks, export_cosmos_road_boundaries, export_cosmos_lane_lines, export_cosmos_traffic_signs, export_cosmos_wait_lines, export_cosmos_road_markings) to emit the byte-compatible HD-map JSON, then renders the HD-map control video via render_hdmap_video(). cosmos_client.py sends the control inputs plus a text prompt to a Cosmos Transfer server and writes back the restyled result.
• Client configuration and assets: AOV configs (cosmos_aov.yaml, sample_configs/, test_configs/), the carla-cosmos-client.yaml conda environment, requirements_client.txt, the semantic-class filter, the process_prompts.sh helper, and a library of TOML scene prompts under example_data/prompts/.
• Cosmos Transfer server packaging under PythonAPI/examples/nvidia/cosmos/server/: Dockerfile.server and make_docker.sh to build the server image, the carla-cosmos-server.yaml conda environment, requirements_server.txt, supervisord.conf, a Gradio app, and the cosmos-transfer1-utils/ deploy/checkpoint/preprocess/transfer-pipeline helpers.
• example_data/ media (committed via Git LFS): the InvertedAI input logs under logs/inverted_ai/ and the reference AOV / HD-map artifacts under artifacts/ (rgb.mp4, semantic_segmentation.mp4, instance_segmentation.mp4, depth.mp4, edges.mp4, hdmap.mp4, vis.mp4). .gitattributes now routes *.log and *.mp4 through LFS.
• cosmos-drive-dreams submodule at PythonAPI/examples/nvidia/cosmos/utils/cosmos-drive-dreams (https://github.com/nv-tlabs/Cosmos-Drive-Dreams.git), which supplies the RDS-HQ rendering utilities used by render_hdmap_video().
• Documentation: the ported guide Docs/nvidia_cosmos_transfer.md (with the Docs/img/cosmos_transfer/brev_01..07.png screenshots), plus the per-component README.md / README_CLIENT.md / README_SERVER.md / RDS_HQ_GENERATION.md. mkdocs.yml gains an "NVIDIA Cosmos Transfer" nav entry under the ecosystem section.
• Smoke test: PythonAPI/test/smoke/test_sensor_tick_time.py now exercises the sensor.camera.cosmos_visualization sensor alongside the other camera variants.

Phase-1 on-PC capture (Task 3.3) — completed

The Phase 1 on-PC end-to-end capture has now been run on the full ue5-dev stack (PR1 #9796 + PR2 #9797 + PR3) on Town10HD_Opt, with a headless editor (-game -RenderOffScreen -nosound -vulkan -carla-rpc-port=2000). The pipeline works end to end:

• Build and launch ue5-dev with PR1 + PR2 + PR3 applied (PR3 adds no C++, so the PR2 editor build serves as the server).
• Run carla_cosmos_gen.py to produce the AOV frames and the six export_cosmos_* HD-map JSONs — all valid RDS-HQ labels/shape3d, with real geometry on Town10HD_Opt (road_boundaries 231, lanelines 168, traffic_signs 162, crosswalks 16, wait_lines 15, road_markings 12; plus poles 120, traffic_lights 163), and 151 frames of camera poses + dynamic-object boxes + pinhole intrinsics.
• Run render_hdmap_video() (the cosmos-drive-dreams toolkit) to render the HD-map control video — lane lines, road boundaries, crosswalks, signs, and dynamic-object boxes project correctly into the ego camera and align with the RGB.
• cosmos_client.py restyle — completed on a single H100; see the comment below for the restyle result and video.
• Screenshots attached below: (a) AOV capture, (b) HD-map control video, (c) restyled result (in the comment below).

Captures

(a) AOV capture — RGB / Depth / Semantic Seg / Instance Seg / Canny Edges / Masked RGB:

(b) HD-map control video from the six export_cosmos_* JSONs — RGB | HD-map | overlay:

Note on the example logs: the bundled example_data/logs/inverted_ai/** were recorded with CARLA 0.9.x and are not replayable on ue5-dev (0.10.0) — show_recorder_file_info reports Frames: 1, Duration: 0 and replay spawns 0 actors (the per-frame recorder packet layout changed between 0.9.x and 0.10.0; the file header still parses). The capture above therefore used a freshly generated 0.10.0 recording on Town10HD_Opt. These example logs are regenerated on ue5-dev so the documented invocation works on this branch.

Part of #9795 (tracking issue: Port NVIDIA Cosmos Transfer support to ue5-dev).

Where has this been tested?

• Platform(s): Linux. The pipeline files, docs build, submodule wiring, LFS media, and smoke-test edit were verified by build/copy on Ubuntu 24.04.4 LTS (kernel 6.17). The CI image is Ubuntu 22.04 (carlasim/carla-builder:ue5-22.04). The full end-to-end capture (Phase-1 on-PC gate above) was run on Ubuntu 24.04 (RTX 5090) on Town10HD_Opt: AOV frames + the six HD-map JSONs + the HD-map control video are verified; the Cosmos Transfer restyle was run on a single H100 (see comment below).
• Python version(s): 3.12 (CARLA Python API target).
• Unreal Engine version(s): 5.5 (CARLA UE5 fork). The pipeline drives the Python API and exporters from PR1/PR2; no further engine changes are introduced by this PR.

Possible Drawbacks

• The example_data/ media and InvertedAI logs are large and committed through Git LFS, so cloning the repo with LFS enabled pulls roughly 166 MB of additional objects.
• Adds the external cosmos-drive-dreams submodule; contributors must run git submodule update --init for render_hdmap_video() to work.
• The end-to-end pipeline requires a GPU host and a reachable Cosmos Transfer server (datacenter-class GPU such as an H100), so it cannot be exercised in CI; correctness of the full run depends on the manual exit gate above.

---

This change is 

[youtalk]

Task 3.3 — Phase-1 exit gate: Cosmos restyle (Step 4c) ✅

The final piece of the Phase-1 on-PC capture gate — the Cosmos Transfer style-conversion — is complete, so the end-to-end pipeline on this PR runs CARLA → control videos → Cosmos Transfer restyle.

Server. A Cosmos Transfer1 server was deployed from the CARLA NVIDIA Brev launchable (carla-x-cosmos-transfer1-lambda) on a single NVIDIA H100 (80 GB). The client connected privately over an SSH port-forward (no public URL). cosmos_client.py talks to the gradio endpoints /upload_file and /generate_video as documented.

Inputs. The three control videos produced earlier on Town10HD_Opt were used unchanged: rgb.mp4 (input), edges.mp4, and semantic_segmentation.mp4 (all 1920×1080 @ 24 fps, 122 frames / ~5 s).

Config. example_data/prompts/rain.toml — edge control weight 0.5, seg control weight 0.9, num_steps=35, guidance=7.0, sigma_max=78, seed=1024.

Result. Cosmos Transfer produced a structure-preserving "steady rain" restyle: the road layout, lane markings, buildings and the lead vehicle are preserved while the scene is transformed to an overcast, wet, photorealistic look (wet road with tail-light reflections, darkened sky). Output video: H.264, 1280×704, 24 fps, 122 frames (~5 s). Generation took ~19 min on the single H100.

CARLA input (left) vs Cosmos restyled (right):

Restyled video:

rain_restyled.mp4

With this, the Phase-1 verification gates for the Cosmos Transfer ue5-dev port are complete (capture + HD-map render + restyle here on #9798; in-editor materials/overlay on #9799).