Gate separation by source presence

Author: Michal Warda

README.md

@@ -72,6 +72,18 @@ curl -f http://localhost:8000/v1/separate \
   -o separation.zip
 ```
 
+The optional `targets` form field accepts `music`, `voice`, or
+`music,voice` (default). A single target runs only one generation stage and the
+ZIP contains that target plus its SFX residual. The durable pipeline sets this
+field from its Audio Flamingo source-scene presence preflight:
+
+```bash
+curl -f http://localhost:8000/v1/separate \
+  -F audio=@mixture.wav \
+  -F targets=voice \
+  -o voice-and-residual.zip
+```
+
 Generation, prompt, batching, and TF32 policy are configured with the
 `SAM_AUDIO_*` environment variables demonstrated in
 [`deploy/start-sam-audio-api.sh`](deploy/start-sam-audio-api.sh). A supplied

docs/pipeline/ARCHITECTURE.md

@@ -18,37 +18,47 @@ rule are persisted under `adaptive_routing`.
 
 1. The control API appends an upload job to a persistent dataset and returns S3 upload URLs. API requests finish
    after durable acceptance; they do not stay open while inference is queued.
-2. An ingestion worker normalizes each source with FFmpeg and creates 30-second
-   chunks with 5-second overlap. The final short chunk is retained.
+2. An ingestion worker probes the source before chunking. Exactly two-channel
+   stereo is required; mono and multichannel sources are durably marked
+   `non_stereo_input` and skipped without model work. Accepted sources are
+   normalized with FFmpeg into 30-second chunks with 5-second overlap. The
+   final short chunk is retained.
 3. A sound gate measures peak and RMS level. Empty or extremely quiet chunks are
    marked `skipped` without consuming GPU time.
-4. Each audible chunk is sent to the single SAM Audio queue. One GPU worker
-   receives one message at a time and calls the local `sam-audio-small-tv` API.
-5. SAM Audio separates `music soundtrack` from the original chunk, then
-   `human voices` from the music residual. Candidate expansion and Judge/CLAP
+4. Audio Flamingo analyzes the complete source before SAM work is scheduled and
+   returns `has_music` and `has_voices`. A malformed response fails open to both
+   targets; a valid negative prevents that separation stage from running.
+5. Each audible chunk is sent to the single SAM Audio queue with zero, one, or
+   two requested targets. Zero targets retains the normalized stereo chunk as a
+   bit-identical SFX pass-through without calling SAM. One target runs one SAM
+   stage. Two targets use the normal cascade.
+6. SAM Audio normally separates `music soundtrack` from the original chunk,
+   then `human voices` from the music residual. Candidate expansion and Judge/CLAP
    evidence determine `success`, `uncertain`, or `failure` independently for
    each stage. A failed complete cascade triggers the bounded reverse-order
    attempt described above; it never recursively separates the SFX residual.
-6. Music, voice, and final residual (SFX) WAVs are retained as the original mono
+   These judges measure candidate quality, not whether the requested sound was
+   present; source-scene preflight owns presence decisions.
+7. Music, voice, and final residual (SFX) WAVs are retained as the original mono
    model outputs. A CPU post-process also creates a stereo companion for every
    audible stem by matching it against the original stereo chunk in 32
-   log-frequency bands. Per-band left/right correlation and loudness are
-   smoothed with a bidirectional EMA (`alpha=0.03`) in time and a three-bin
-   frequency kernel before equal-power panning. Only a final peak limiter is
-   allowed to alter the reconstructed level.
-7. Raw and stereo-mapped WAVs, the 120-point pan/loudness trajectory, inference
+   log-frequency bands. Per-band left/right correlation controls panning, while
+   loudness transfer is broadband so it cannot re-EQ the stem. Both are smoothed
+   with a bidirectional EMA (`alpha=0.03`) before equal-power panning. An SFX-only
+   pass-through bypasses mapping and keeps the original stereo PCM exactly.
+8. Raw and stereo-mapped WAVs, the 120-point pan/loudness trajectory, inference
    metadata, timings, and review assertions are stored in S3 and indexed in
    DynamoDB. A stereo mapping failure never discards or delays the raw result.
-8. Description and transcription tasks share one Audio Flamingo queue so that a
+9. Description and transcription tasks share one Audio Flamingo queue so that a
    single loaded model processes one item at a time. It creates one source-scene
    analysis per audible source, one description per music stem, and one
    speaker-labelled transcript per voice stem. Tasks exist in DynamoDB before
    SQS delivery and are reconciled after worker/message loss.
-9. A browser dashboard shows job/queue progress and exact remaining review
+10. A browser dashboard shows job/queue progress and exact remaining review
    counts. Its reviewer mode pulls the next `uncertain` or `failure` assertion,
    starts audio automatically, displays the prompt/assertion, and accepts
    one-key decisions.
-10. The dataset explorer reports source count, duration, input/chunk/stem storage,
+11. The dataset explorer reports source count, duration, input/chunk/stem storage,
    processing counts, stem mix, and review backlog. Each source displays the
    original recording first, then only the music/voice/SFX stems that passed the
    output sound gate, grouped by chunk, followed by its processing route and

docs/pipeline/OPERATIONS.md

@@ -58,6 +58,15 @@ than keep an inference request open.
 
 ## Stereo mapping
 
+The pipeline accepts exactly two-channel stereo sources. Mono and multichannel
+uploads remain visible in the dataset but finish with `skip_reason` set to
+`non_stereo_input`; they create no chunks, stems, or model tasks.
+
+For accepted sources, Audio Flamingo scene analysis runs before SAM. Its
+`has_music` and `has_voices` booleans select zero, one, or both SAM targets. A
+malformed scene response deliberately falls back to both targets. Judge/CLAP
+scores remain separation-quality evidence and are not used as presence scores.
+
 New audible stems automatically keep both versions:
 
 - `{stem}.wav` is the untouched mono model output used by review and existing
@@ -69,6 +78,10 @@ New audible stems automatically keep both versions:
 In **Split data**, select a source and use **Raw / Stereo mapped** above the
 players. The original player never changes. The maps below the players show the
 smoothed trajectory; left is below the center line and right is above it.
+Version 2 applies frequency-specific panning but only broadband gain, preventing
+the mapper from making stems bass-heavy. A source with neither music nor voices
+is stored as a single SFX stem; Raw and Stereo mapped are PCM-identical and the
+map is labelled `identity · no EQ`.
 
 Backfill a completed job after deploying the mapper. The command is idempotent
 and skips stems that already have a mapped object unless `--force` is supplied:

docs/pipeline/PROGRESS.md

@@ -32,6 +32,8 @@ was an example batch size, not a workflow limit.
   original-plus-stems source explorer.
 - [x] Add frequency-aware stereo/loudness reconstruction while preserving every
   raw mono stem, plus a dashboard playback toggle and smoothed trajectory plots.
+- [x] Require stereo input, gate SAM targets with source-scene presence, support
+  single-stage inference, and use identity pass-through for pure SFX sources.
 - [x] Add persistent datasets, successive upload jobs, and reconciliation of
   uploaded/stale work from DynamoDB back into SQS.
 - [x] Provision AWS, deploy services, and run a multi-file end-to-end test.
@@ -60,17 +62,20 @@ was an example batch size, not a workflow limit.
   191,551,793 bytes.
 - Dataset `991945c1f082446a9ff66d482838f964`, job
   `3f0233f2442f4044a0f2967c68cb5a89`, completed all 100 sources and chunks with
-  zero failed chunks. It stored 284 audible stems (94 music, 91 voice, 99 SFX)
+  zero failed chunks. It initially stored 284 audible stems (94 music, 91 voice,
+  99 SFX)
   and omitted 16 below-gate outputs (6 music, 9 voice, 1 SFX). The selected
   route was music-first for 63 sounds and voice-first for 37.
-- The completed random dataset contains 624.4 MiB of indexed originals,
-  normalized chunks, and raw stems plus 543,877,704 bytes of mapped stereo
-  companions. All 284 audible stems were backfilled without replacing their
-  raw keys. Automatic verification placed 161 stems in
-  success, 54 in uncertain, and 69 in failure, leaving 123 review items.
+- After the presence repair, the completed random dataset contains 282 stems
+  and 540,037,650 bytes of mapped stereo companions. All 282 companions were
+  force-migrated to stereo v2 (or the identity path) without replacing their
+  raw keys. Automatic verification places 159 stems in success, 54 in
+  uncertain, and 69 in failure, leaving 123 review items.
 - Browser verification confirms the review screen shows exact remaining,
   failure, and uncertain counts; the dataset screen shows 100 sounds and its
-  storage breakdown; and gated stems are absent from the stacked track view.
+  storage breakdown; gated stems are absent from the stacked track view; the
+  repaired writing source shows `sfx only` plus `identity · no EQ`; and the mono
+  canary shows the explicit stereo-input skip reason.
 - Live reference job `ae31001eec754ee6850e3727134de84e` processed the supplied
   30-second `chunk_ours/og.wav` through the normal production handler and
   created raw plus stereo music, voice, and SFX objects. The rendered dashboard
@@ -81,6 +86,17 @@ was an example batch size, not a workflow limit.
   (right), matching the attached chunk-zero analysis. All three 30-second stems
   map in about 0.58 seconds locally; the fresh remote job mapped its three stems
   in about 1.02 seconds.
+- Source `099_writing_mHf7COLGcD0_170.wav` exposed both failure modes: its scene
+  annotation correctly said no music and no voices, but the old cascade still
+  created three `success` stems, and stereo v1 reduced music high-frequency
+  energy from 8.7% to 0.8%. Stereo v2 retains 7.8% on the same raw stem. The live
+  source was repaired from its stored scene evidence and now has one SFX-only
+  stem; raw, mapped, and original normalized PCM are sample-identical. The
+  dataset now indexes 282 stems (93 music, 90 voice, 99 SFX).
+- Live mono-filter job `5ec26b1bb28b420a9245e21d475d26be` completed with
+  `non_stereo_input`, one input channel, zero chunks, zero stems, and zero model
+  tasks. A live `targets=voice` API canary produced only voice and residual
+  artifacts in one 12.8-second SAM stage.
 - Multi-file job `bd85486f9d314c258956cac9b81a730f` completed: one 30-second
   audible source produced exactly one chunk and three stems; one silent source
   was sound-gated without GPU work; all three Audio Flamingo tasks completed.
@@ -90,7 +106,7 @@ was an example batch size, not a workflow limit.
 - Recovery job `1d07979ee96a45a98cf220b102fd7006` was uploaded without the
   completion callback. Reconciliation found the S3 object, enqueued ingestion,
   and completed it as sound-gated. All three queues were empty afterward.
-- Local and remote validation: 28 pipeline tests pass; the existing 12 SAM
+- Local and remote validation: 36 pipeline tests pass; the existing 12 SAM
   batching tests also pass.
 - Docker build execution remains unverified because neither the local Mac nor
   the current H100 host has a Docker daemon. Both Dockerfiles remain model-free.

pipeline/src/sam_audio_pipeline/api.py

@@ -173,6 +173,10 @@ def _dataset_snapshot(aws: PipelineAWS, dataset_id: str) -> dict[str, Any]:
         "summary": {
             "jobs": len(jobs),
             "sources": len(all_sources),
+            "non_stereo_sources": sum(
+                source.get("skip_reason") == "non_stereo_input"
+                for source in all_sources
+            ),
             "duration_seconds": sum(
                 float(source.get("duration_seconds") or 0.0) for source in all_sources
             ),

pipeline/src/sam_audio_pipeline/audio.py

@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 import hashlib
+import json
 import math
 import subprocess
 import wave
@@ -63,6 +64,28 @@ def probe_duration(path: Path) -> float:
     return duration
 
 
+def probe_channels(path: Path) -> int:
+    result = _run(
+        [
+            "ffprobe",
+            "-v",
+            "error",
+            "-select_streams",
+            "a:0",
+            "-show_entries",
+            "stream=channels",
+            "-of",
+            "json",
+            str(path),
+        ]
+    )
+    streams = json.loads(result.stdout).get("streams", [])
+    channels = int(streams[0].get("channels") or 0) if streams else 0
+    if channels <= 0:
+        raise ValueError(f"Audio channel count is unavailable for {path}")
+    return channels
+
+
 def gate_wav(
     path: Path,
     *,