feat(ai-gemini): Gemini Omni Flash video generation via the Interactions API (#886)

* feat(ai-gemini): support Gemini Omni Flash video generation via the Interactions API

Add gemini-omni-flash-preview to the Gemini video adapter. Omni only
serves the Interactions API (generateContent rejects it with 400), so
the adapter now routes by model: Veo models keep the
:predictLongRunning operations flow, while Omni creates a background
interaction with response_modalities: ['video'], polls it by id, and
returns the inline base64 MP4 as a data: URL (Files-API URI delivery
passes through). Usage maps from output_tokens_by_modality, size maps
onto response_format.aspect_ratio, and
modelOptions.previous_interaction_id chains conversational video edits.

- model-meta: GEMINI_OMNI_FLASH_PREVIEW ($0.10/sec video+audio output)
  + GEMINI_INTERACTIONS_VIDEO_MODELS
- provider options: GeminiOmniVideoProviderOptions derived from the
  SDK's CreateModelInteractionParamsNonStreaming; per-model input
  modalities (Omni accepts image+video parts) and fixed 10s duration
- @google/genai floor bumped to ^2.10.0 for the interactions surface
- 17 new unit tests; new interactions-video E2E feature backed by a
  dedicated aimock mount (native interactions text handling untouched)
- docs/media/video-generation.md + media-generation skill updates

Verified live against the Gemini API: background job completed in ~45s
and returned a valid MP4 with video-modality usage; the SDK's typed
interactions.create works with Step-list input, so no raw REST
fallback is needed.

Closes #871

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(examples): Gemini Omni Flash with all inputs in ts-react-media

Add gemini-omni-flash-preview (text-to-video + image-to-video) to the
ts-react-media example, exercising every Omni input: text prompts, a
start image, an attached reference/edit video clip (Omni-only — never
sent to other providers), and conversational editing that chains a new
prompt onto a completed generation via previous_interaction_id.

Also fixes a latent core type bug this surfaced: generateVideo /
getVideoJobStatus constrained adapters as VideoAdapter<string, any,
any, any>, leaving the duration generic at its Record<string, number>
default — any adapter with a narrowed per-model duration union (Omni's
10, Veo's 4|6|8) failed assignability under strict function-type
contravariance. All video-activity constraints now span all six
VideoAdapter generics.

Verified live: Omni edit chaining (previous_interaction_id) against the
real Gemini API returned an edited 10s MP4; example dev server boots
and type-checks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(ai-gemini): support Omni Flash 3-10s clip durations

The issue's live verification concluded Omni clips were a fixed 10
seconds, but response_format.duration is a real request field — just
undocumented. Verified against the live API: it takes a "<seconds>s"
string, accepts any value in the 3-10s range including fractional
seconds (a 3s request returns a 3.008s MP4 per ffprobe), rejects
out-of-range values with explicit minimum/maximum errors, and defaults
to 10s when omitted.

Omni's duration is now typed number with availableDurations() =
{ kind: 'range', min: 3, max: 10, unit: 'seconds' } and snapDuration
clamping into it; the adapter maps the generateVideo duration option
onto response_format.duration.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ai-gemini): address PR review findings for Omni Flash video

- Reject out-of-range Omni durations at job creation with a clear local
  error instead of silently passing them to the live API
- Map requires_action interactions to a failed status so polling can't
  spin until timeout (reachable via previous_interaction_id chaining)
- Surface failed job statuses in the ts-react-media example instead of
  polling forever on a pending spinner
- Add a compile-time regression test guarding the generateVideo
  VideoAdapter generic-arity fix, plus unit tests for duration
  rejection, fractional pass-through, and requires_action mapping
- Fix stale doc/comment claims: Veo 2/3 model lists, "fixed 10s" clips,
  "clamped" duration wording, and content-block ordering (images, then
  videos, then text)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(examples): Omni Studio conversational session view in ts-react-media

Add a chat-style session tab built around Gemini Omni Flash's
conversational editing:

- Timeline of turns where each completed clip auto-becomes the
  continue-from target; "Continue from here" branches the session from
  any earlier clip via previous_interaction_id
- Multi-reference image tray (Omni supports ~6), single reference clip
  slot labeled with the 3s API limit, and a prompt-steered audio
  direction field
- Per-turn duration slider (3-10s) doubling as a $0.10/s cost estimate,
  aspect toggle, and task-mode pin mapped to
  generation_config.video_config.task
- Duration is locked and omitted for edit-shaped requests (continuing a
  clip, task=edit, or attached clip without a reference pin) — Omni
  rejects durations that don't match the source clip's length
- Failed turns get an "Edit & retry" button that restores the prompt,
  attachments, and settings to the composer and removes the dead turn

createVideoJobFn gains omniOptions (duration / aspectRatio / task) to
carry the new controls.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Author: tombeckenham

.changeset/gemini-omni-flash-video.md

@@ -0,0 +1,5 @@
+---
+'@tanstack/ai-gemini': minor
+---
+
+Add Gemini Omni Flash (`gemini-omni-flash-preview`) video generation via the Interactions API. Omni only serves the Interactions API (`generateContent` rejects it), so the video adapter now routes by model: Veo models keep the `:predictLongRunning` operations flow, while `geminiVideo('gemini-omni-flash-preview')` creates a background interaction with `response_modalities: ['video']`, polls it by id, and returns the inline base64 MP4 as a `data:` URL (Files-API URI delivery passes through). Usage is mapped from the interaction's `output_tokens_by_modality`. Image and video prompt parts are sent as interaction content blocks, and `modelOptions.previous_interaction_id` chains a new prompt onto a prior Omni generation for conversational video editing. The top-level `size` option maps onto `response_format.aspect_ratio` (`'16:9' | '9:16'`) and `duration` onto `response_format.duration` — any value in the 3–10 second range (fractional seconds included, verified against the live API), defaulting to a 10-second clip when omitted. Raises the `@google/genai` floor to `^2.10.0` for the Interactions API surface.

.changeset/video-adapter-duration-constraint.md

@@ -0,0 +1,5 @@
+---
+'@tanstack/ai': patch
+---
+
+Fix `generateVideo` / `getVideoJobStatus` rejecting video adapters that declare a narrowed per-model duration union (e.g. Gemini's `4 | 6 | 8` for Veo or `10` for Omni Flash) at the type level. The activity's `TAdapter extends VideoAdapter<string, any, any, any>` constraints left the input-modality and duration generics at their defaults, so `duration?: number` failed contravariance against the adapter's literal union. All video-activity constraints and helper conditionals now span all six `VideoAdapter` generics.

docs/config.json

@@ -294,7 +294,7 @@
           "label": "Video Generation",
           "to": "media/video-generation",
           "addedAt": "2026-04-15",
-          "updatedAt": "2026-07-01"
+          "updatedAt": "2026-07-02"
         },
         {
           "label": "Generation Hooks",

docs/media/video-generation.md

@@ -2,12 +2,14 @@
 title: Video Generation
 id: video-generation
 order: 6
-description: "Generate video from text prompts with OpenAI Sora, Google Veo, xAI Grok Imagine, or fal.ai using TanStack AI's experimental generateVideo() jobs/polling API."
+description: "Generate video from text prompts with OpenAI Sora, Google Veo, Gemini Omni Flash, xAI Grok Imagine, or fal.ai using TanStack AI's experimental generateVideo() jobs/polling API."
 keywords:
   - tanstack ai
   - video generation
   - sora
   - veo
+  - omni flash
+  - interactions api
   - gemini
   - grok imagine
   - fal
@@ -40,7 +42,7 @@ TanStack AI provides experimental support for video generation through dedicated
 
 Currently supported:
 - **OpenAI**: Sora-2 and Sora-2-Pro models (when available)
-- **Google Gemini**: Veo 3.1, Veo 3, and Veo 2 models (via the long-running operations API)
+- **Google Gemini**: Veo 3.1 models (via the long-running operations API), and Gemini Omni Flash (via the Interactions API)
 - **Grok (xAI)**: grok-imagine-video (text-to-video + image-to-video) and grok-imagine-video-1.5 (image-to-video only) models
 - **fal.ai**: MiniMax, Luma, Kling, Hunyuan, and other hosted video models
 
@@ -569,6 +571,85 @@ Adapters that haven't declared a per-model duration map keep the plain
 > Files API and requires your API key to download (send it as an
 > `x-goog-api-key` header or `key` query parameter).
 
+### Gemini Omni Flash (Interactions API) Model Options
+
+Gemini Omni Flash (`gemini-omni-flash-preview`) is Google's multimodal
+video-generation model with conversational editing. It only serves the
+[Interactions API](https://ai.google.dev/gemini-api/docs/omni) — the same
+`geminiVideo()` adapter routes it automatically: `generateVideo` creates a
+background interaction, `getVideoJobStatus` polls it by id, and the
+finished clip comes back **inline as a `data:video/mp4;base64,…` URL**
+(when Google delivers by reference instead, the Files API URI passes
+through and needs your API key to download, like Veo).
+
+Clips are 720p at 24 FPS, and `duration` accepts any value in the **3–10
+second** range (fractional seconds included), defaulting to 10 seconds when
+omitted. `availableDurations()` reports
+`{ kind: 'range', min: 3, max: 10, unit: 'seconds' }`; out-of-range
+`duration` values are rejected at job creation, and `snapDuration(n)` snaps
+raw seconds into the range (clamping to its bounds and rounding to whole
+seconds). The `size` option maps onto the interaction's output aspect
+ratio:
+
+```typescript ignore
+import { generateVideo, getVideoJobStatus } from '@tanstack/ai'
+import { geminiVideo } from '@tanstack/ai-gemini'
+
+const adapter = geminiVideo('gemini-omni-flash-preview')
+
+const { jobId } = await generateVideo({
+  adapter,
+  prompt: 'A woman playing violin outdoors at golden hour',
+  size: '9:16', // aspect ratio: '16:9' (default) or '9:16'
+  duration: 6, // 3-10 seconds; omit for the 10s default
+})
+
+const status = await getVideoJobStatus({ adapter, jobId })
+// status.url → 'data:video/mp4;base64,…' once completed
+```
+
+Image and video prompt parts are sent to the interaction as content blocks
+— grouped as images, then videos, then the text prompt (Omni doesn't use
+Veo's `metadata.role` routing) — so you can condition the generation on
+stills or short reference clips. `data` sources
+are sent inline as base64; `url` sources pass through as-is — the adapter
+never downloads them, so use Gemini Files API URIs (upload large media via
+the Files API first).
+
+#### Conversational video editing
+
+Omni's headline capability is iterative refinement: pass the interaction id
+of a prior generation (its `jobId`) as
+`modelOptions.previous_interaction_id` and describe the change — the model
+edits the video while preserving everything you didn't mention:
+
+```typescript ignore
+import { generateVideo } from '@tanstack/ai'
+import { geminiVideo } from '@tanstack/ai-gemini'
+
+const adapter = geminiVideo('gemini-omni-flash-preview')
+
+// Turn 1: generate
+const first = await generateVideo({
+  adapter,
+  prompt: 'A woman playing violin outdoors at golden hour',
+})
+
+// …poll first.jobId to completion, then…
+
+// Turn 2: edit the result conversationally
+const second = await generateVideo({
+  adapter,
+  prompt: 'Make the violin invisible',
+  modelOptions: { previous_interaction_id: first.jobId },
+})
+```
+
+`modelOptions` also passes through the Interactions API's request fields
+(e.g. `generation_config.video_config.task` to pin
+`'text_to_video' | 'image_to_video' | 'reference_to_video' | 'edit'`
+instead of letting the model infer the task mode).
+
 ### Grok (xAI Imagine) Model Options
 
 Based on the [xAI video generation API](https://docs.x.ai/docs/guides/video-generations). Two models are available: `grok-imagine-video` (v1.0) supports **text-to-video and image-to-video**, while `grok-imagine-video-1.5` is **image-to-video only** (a text-only prompt is rejected by the API; the adapter throws a clear error pointing you at `grok-imagine-video`). Both are aspect-ratio sized — the generic `size` option takes an `aspectRatio_resolution` template (like the Grok Imagine image models), and clips can be 1–15 seconds long.

examples/ts-react-media/src/components/ImageGenerator.tsx

@@ -6,8 +6,8 @@ import type { MediaPrompt } from '@tanstack/ai/client'
 import { generateImageFn } from '@/lib/server-functions'
 import { getRandomImagePrompt } from '@/lib/prompts'
 import { IMAGE_MODELS } from '@/lib/models'
-import { readImageFile, toImagePart } from '@/lib/media'
-import type { AttachedImage } from '@/lib/media'
+import { readMediaFile, toImagePart } from '@/lib/media'
+import type { AttachedMedia } from '@/lib/media'
 
 interface ImageGeneratorProps {
   onImageGenerated?: (imageUrl: string) => void
@@ -36,7 +36,7 @@ export default function ImageGenerator({
   const [selectedModel, setSelectedModel] = useState<string>('all')
   const [isLoading, setIsLoading] = useState(false)
   const [results, setResults] = useState<Record<string, ModelResult>>({})
-  const [images, setImages] = useState<Array<AttachedImage>>([])
+  const [images, setImages] = useState<Array<AttachedMedia>>([])
   const fileInputRef = useRef<HTMLInputElement>(null)
 
   const currentModel = IMAGE_MODELS.find((m) => m.id === selectedModel)
@@ -56,7 +56,7 @@ export default function ImageGenerator({
     const files = Array.from(e.target.files ?? [])
     if (fileInputRef.current) fileInputRef.current.value = ''
     if (files.length === 0) return
-    const attached = await Promise.all(files.map((file) => readImageFile(file)))
+    const attached = await Promise.all(files.map((file) => readMediaFile(file)))
     setImages((prev) => [...prev, ...attached])
   }