AI agents that live inside photorealistic 3D Gaussian splat worlds. Walk through immersive environments, overhear autonomous agent conversations, and join the dialogue in real time. Agents can reshape the world around you mid-conversation.
Oracle Society places AI agents inside photorealistic 3D environments rendered with Gaussian splatting. Each world — a Manhattan streetscape, an ancient temple, a cyberpunk rooftop, a haunted house, a cozy cottage, a spaceship interior — is a pre-generated 3D scene hosted on CDN. Agents inhabit these worlds as glowing entities positioned in 3D space, visible as colored orbs with floating name labels.
When you enter a world, agents are already mid-conversation. As you navigate closer using arrow keys, WASD, scroll wheel, or head tracking via your webcam, their voices fade in based on proximity. At a distance you hear faint murmurs; up close, full dialogue. Each agent has a distinct personality, role, and voice — a philosopher debating consciousness in a temple, a hacker discussing digital art on a cyberpunk rooftop, an economist arguing about urban infrastructure on Fifth Avenue.
Press Space to join the conversation. Agents acknowledge your presence naturally and respond to your voice in real time through Gemini Live's WebSocket connection. During the conversation, an agent might suggest a scene change — "Let me take you somewhere that fits this thought..." — and the entire environment smoothly transitions to a different world while agents remain in place. The result is an experience where AI agents don't live in chat windows. They live in worlds. You visit them.
- Three.js + Spark (@sparkjsdev/spark) — Gaussian splat 3D rendering engine
- Gemini 2.0 Flash Live — Real-time bidirectional voice conversation via WebSocket with function calling
- Gemini 2.0 Flash REST — Autonomous dialogue generation between agents
- MiniMax APIs — Text-to-speech (speech-2.8-hd), image generation, music generation, video generation via Express proxy
- World Labs Marble — 3D Gaussian splat world generation (pre-generated, cached on CDN)
- MediaPipe FaceMesh — Head tracking for hands-free view rotation
- Vite 6 — Dev server and build tooling
- Express 5 — Backend proxy for MiniMax API calls
- Web Speech API — Browser-native speech recognition and TTS fallback
- Vanilla JS — No frontend framework, direct Three.js and DOM manipulation
BROWSER
┌──────────────────────────────────────────────────────┐
│ │
│ Three.js + Spark Agents Layer │
│ Gaussian Splat Render Positioned in 3D space │
│ Head Tracking Proximity detection │
│ Arrow Keys / WASD Volume scaling │
│ │
│ Gemini Live (WebSocket) │
│ Real-time voice chat │
│ Function calling: │
│ - change_world() │
│ - fly_to() │
│ - generate_image() │
│ - play_music() │
│ │
│ providers.js │
│ Gemini ↔ MiniMax toggle │
└──────────────────────┬───────────────────────────────┘
│ HTTP
▼
┌────────────────┐
│ server.js │
│ Express proxy │
│ Port 3456 │
│ │
│ /api/tts │
│ /api/image │
│ /api/video │
│ /api/music │
│ /api/voices │
└────────┬───────┘
│
▼
MiniMax Cloud APIs
Six pre-generated photorealistic environments, each with two resident AI agents:
| World | Agents | Theme |
|---|---|---|
| Manhattan Fifth Ave | Nova (Urban Strategist) + Atlas (Behavioral Economist) | Cities, infrastructure, AI urbanism |
| Ancient Temple | Sage (Philosopher of Mind) + Echo (Historian) | Consciousness, ethics, cycles of history |
| Cyberpunk Rooftop | Cipher (Hacker) + Pixel (Digital Artist) | Digital art, surveillance, agent societies |
| Haunted House | — | Fear, the unknown |
| Cozy Cottage | — | Home, comfort, belonging |
| Spaceship | — | The future, space, post-humanity |
Agent audio and interaction scale with your distance from them:
| Zone | Distance | Volume | Behavior |
|---|---|---|---|
| Far | > 12 units | 0% | Silent |
| Murmur | 7 - 12 | 15% | Faint background chatter |
| Clear | 3 - 7 | 60% | Full conversation audible |
| Join | < 3 | 100% | "Press Space to join" prompt appears |
oracle-society/
├── index.html UI shell — chat panel, agent HUD, transition overlay
├── server.js Express proxy for MiniMax APIs (port 3456)
├── generate.js World Labs Marble API — offline 3D world generator
├── package.json
├── src/
│ ├── main.js Three.js renderer, controls, keyboard walking, animate loop
│ ├── gemini.js Gemini Live WebSocket — voice chat + function calling
│ ├── gestures.js MediaPipe FaceMesh head tracking (view rotation)
│ ├── agents.js Agent lifecycle — spawn/despawn per world, dialogue state
│ ├── agent-entity.js 3D glowing orb + floating name label per agent
│ ├── dialogue.js Dialogue generation (Gemini REST) + TTS playback engine
│ ├── proximity.js Camera-to-agent distance zones, volume scaling
│ ├── providers.js Provider toggle — Gemini (browser TTS) ↔ MiniMax
│ ├── minimax.js MiniMax API client (TTS, image, music via server proxy)
│ ├── worlds.js World configs — splat URLs, agent positions, dialogue seeds
│ ├── integration.js Glue layer — wires agents, proximity, transitions together
│ ├── audio.js Ambient audio (oscillator drone + music playback)
│ ├── particles.js Particle effects
│ ├── postfx.js Post-processing effects
│ └── entity.js Original morphing entity (reference)
└── worlds/ Pre-generated world JSON metadata
- Node.js 18+
- A Gemini API key (entered in browser at runtime)
- A MiniMax API key (optional, for enhanced TTS/image/music — set in
.env)
# Install dependencies
npm install
# Start the MiniMax proxy server (optional)
node server.js
# Start the Vite dev server
npm run devOpen http://localhost:5173. Enter your Gemini API key when prompted.
- Arrow keys / WASD — Walk
- Mouse drag — Look around
- Scroll wheel — Move forward / back
- Head tracking — Toggle via "head track" button (uses webcam)
- Space — Join nearby agent conversation
- Escape — Leave conversation
WLT_API_KEY=your_key node generate.js "A rooftop garden overlooking Tokyo at sunset"World metadata is saved to worlds/. Splat URLs are hosted on World Labs CDN.
MINIMAX_API_KEY=your_minimax_key # .env file, read by server.js
The Gemini API key is entered directly in the browser and never touches the server.
Agents live in worlds, not chat windows. The core insight is spatial presence. Walking toward a conversation and overhearing it before joining creates a fundamentally different relationship with AI than clicking a text box.
Gemini Live for real-time, MiniMax for pre-rendered. Gemini's bidirectional WebSocket handles live voice conversation. MiniMax slots in for the high-quality pre-rendered content: distinct agent voices, ambient world music, and generated images. The providers.js abstraction makes switching a one-line change.
Zero runtime World Labs calls. All six 3D environments are pre-generated Gaussian splats cached on CDN. The generate.js script exists for offline world creation only, preserving API credits and eliminating load-time generation delays.
Proximity-based audio. Volume scales continuously with distance. This makes the 3D space feel inhabited rather than scripted — you discover conversations by exploring.
TRAE x MiniMax NYC Hackathon -- March 30, 2026
MIT