Kimodo Motion — text-to-motion on fal, rendered in 3D

Generate human motion from a text prompt with NVIDIA Kimodo, served on fal serverless (H100), and watch it play back in the browser — either as the raw SOMA body mesh or retargeted onto a rigged 3D character standing in a low-poly world.

• Text → motion via Kimodo motion diffusion, streamed with live denoising progress (SSE).
• Two render modes: the generated SOMA body mesh, or a rigged avatar driven by the motion.
• Character picker: several bundled avatars (ReadyPlayerMe + Mixamo), swappable live.
• Swappable worlds: a procedural low-poly forest (terrain, trees, rocks, fog, sun), a neutral studio, and a dark void — all fully procedural, no assets or network.

serverless/app.py  (fal, H100)                 frontend/  (Next.js + react-three-fiber)
──────────────────────────────                 ─────────────────────────────────────────
prompt ─▶ Kimodo diffusion (SOMA skeleton)      SSE stream ─▶ decode
       ─▶ SOMASkin body mesh  ─┐                            ├─▶ SOMA body mesh, or
       ─▶ per-frame joint       ├─ SSE ─▶ /api/generate ────┤
          rotations + positions ┘                           └─▶ retarget onto rigged avatar

Prerequisites

• Node.js 18+ (20+ recommended)
• Python 3.10+
• A fal account (fal auth login) — the backend runs on a fal GPU (H100).

Backend (fal serverless)

cd serverless
python -m venv .venv && source .venv/bin/activate
pip install -e .            # installs the fal CLI + deps
fal auth login             # one-time

# Run it (ephemeral dev endpoint — keep this process alive):
fal run --auth shared app.py::KimodoApp
#   ...or deploy a persistent endpoint:
# fal deploy

fal run/fal deploy prints a Sync URL ending in /generate — you'll need it below. First run builds the container image and downloads model weights (a few minutes); later runs reuse the warm runner. Then create an API key for the frontend:

fal keys create --scope API --desc kimodo    # prints  FAL_KEY=<id>:<secret>

Note: fal run is ephemeral — the endpoint lives only while that process runs. Use fal deploy for a persistent URL that survives restarts.

Frontend

cd frontend
npm install
npm run fetch-avatars                 # downloads the demo avatar GLBs into public/
cp .env.example .env.local            # then edit .env.local with your FAL_URL + FAL_KEY
npm run dev                           # http://localhost:3000

.env.local (never committed):

FAL_URL=https://fal.run/<user>/<app-id>/generate
FAL_KEY=<key-id>:<key-secret>

Usage

1. Type a motion prompt (e.g. "a person dancing energetically") and Generate.
2. Top-right: pick a World (Forest / Studio / Void) and toggle Body mesh ↔ Character.
3. In Character mode, pick an avatar from the character row.

Tips: stationary prompts (dance, wave, jumping jacks) stay centered; "walking forward" uses real root motion and strolls across the scene.

Adding your own character

Any Mixamo- or ReadyPlayerMe-rigged humanoid GLB works (bone names are matched prefix/case-insensitively, and retargeting is direction-based so any bind pose is fine). Add an entry to CHARACTERS in frontend/app/lib/constants.ts:

{ id: "me", label: "Me", url: "https://models.readyplayer.me/<your-id>.glb" }

It loads client-side and appears in the picker. Local files in public/ work too (url: "/my.glb").

How retargeting works

Kimodo outputs motion on its 77-joint SOMA skeleton (per-frame global joint rotations + world positions), which the backend streams. The frontend maps SOMA joints → the avatar's bones by name and aims each bone along SOMA's live world-space limb direction, orienting the pelvis from a hip/spine/thigh position basis. Working in directions (not raw quaternions) makes it robust to differing bone-axis conventions and bind poses across avatars.

Credits

• Motion model: NVIDIA Kimodo
• Serverless GPU: fal
• Demo avatars/rigs: three.js example models (Mixamo, ReadyPlayerMe)