Skip to content

Commit 34d9613

Browse files
author
crispasr integration
committed
fix(bindings): expose FoxNose speaker turns
1 parent a12e789 commit 34d9613

12 files changed

Lines changed: 406 additions & 53 deletions

File tree

.github/workflows/regression.yml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,8 @@ on:
4444
pull_request:
4545
paths:
4646
- 'tests/regression/**'
47+
- 'tests/test_python_diarize_turns.py'
48+
- 'python/crispasr/**'
4749
- '.github/workflows/regression.yml'
4850

4951
permissions:
@@ -98,6 +100,10 @@ jobs:
98100
run: |
99101
python -m pytest tests/test_python_chat.py -v || \
100102
python -m unittest tests/test_python_chat.py -v
103+
- name: Run Python diarization ABI tests
104+
run: |
105+
python -m pytest tests/test_python_diarize_turns.py -v || \
106+
python tests/test_python_diarize_turns.py -v
101107
# Perf-regression compare logic (docs/perf-sweep/PLAN.md TODO-4). Model-free —
102108
# guards the hard/soft-gate logic used by the nightly perf snapshot step below.
103109
- name: Run perf-baseline compare unit tests

PLAN.md

Lines changed: 7 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -618,14 +618,13 @@ verified locally on `samples/multispeaker.wav` + wespeaker-resnet34-lm, where
618618
the fixture does exercise a segment covering two speakers. Same pair of
619619
levels on the Rust side in `crispasr/tests/integration.rs`.
620620

621-
**Not done, and deliberately: the other bindings.** Go, Python, Java, Ruby,
622-
Dart and JS still expose only `crispasr_diarize_segments_abi`. Nothing there
623-
is broken — the new symbol is additive — but a caller on those surfaces still
624-
cannot split a segment. Extending them is a mechanical follow-up (each has a
625-
hand-written mirror; the new turn struct is its own 24-byte POD, NOT an
626-
append to the opts struct). `tests/test_binding_parity.py`'s curated symbol
627-
list is unchanged for the same reason: adding the symbol there would fail
628-
until the Python binding declares it.
621+
Follow-up completed 2026-09-06: Go, Python, and Dart now size and retry the
622+
turn buffer and expose typed turn results; Java declares both calls on its
623+
low-level JNA surface. The Python audit also found and repaired a separate
624+
ABI bug: its options mirror had remained at 24 bytes after FoxNose extended
625+
the native append-only struct to 48 bytes. JavaScript and Ruby have no public
626+
standalone diarization API today (their unused C declarations were previously
627+
mistaken for binding support), so there is no turn-returning surface to extend.
629628

630629
## CLAIMED 2026-08-19 — Issue #375 Canary streaming regression
631630

bindings/java/src/main/java/io/github/ggerganov/whispercpp/CrispasrSession.java

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -315,6 +315,9 @@ int crispasr_pyannote_cache_apply_abi(Pointer cache, long sliceT0Cs,
315315
// Diarization
316316
int crispasr_diarize_segments_abi(float[] leftPcm, float[] rightPcm, int nSamples,
317317
int isStereo, Pointer segs, int nSegs, Pointer opts);
318+
int crispasr_diarize_segments_turns_abi(float[] leftPcm, float[] rightPcm, int nSamples,
319+
int isStereo, Pointer segs, int nSegs, Pointer opts,
320+
Pointer outTurns, int turnsCap, IntByReference outNTurns);
318321

319322
// Text-LID
320323
int crispasr_text_detect_language(String text, String modelPath, int nThreads,

docs/bindings.md

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -173,7 +173,8 @@ exposes them as struct/class members on its segment type.
173173

174174
```python
175175
from crispasr import (
176-
Session, diarize_segments, detect_language_pcm,
176+
Session, diarize_segments, diarize_segments_with_turns,
177+
detect_language_pcm,
177178
align_words, cache_ensure_file, registry_default_bundle,
178179
# Diarize pipeline primitives (#107):
179180
SpeakerEmbedder, PyannoteCache, agglomerative_cluster,
@@ -188,6 +189,10 @@ segs = sess.transcribe_vad(pcm, "silero-v6.2.0.bin") # stitched VAD pass
188189
# Run each shared post-step standalone
189190
lang = detect_language_pcm(pcm, model_path="ggml-tiny.bin")
190191
diarize_segments(my_segs, pcm, method=DiarizeMethod.VAD_TURNS)
192+
ok, turns = diarize_segments_with_turns(
193+
my_segs, pcm, method=DiarizeMethod.FOXNOSE,
194+
foxnose_embedder_path="wespeaker-resnet34-lm.gguf",
195+
)
191196
words = align_words("canary-ctc-aligner.gguf", "hello world", pcm)
192197

193198
# Inspect the canonical bundle used by `-m auto` (no quant suffix).

docs/diarization-speakers.md

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -298,8 +298,11 @@ for them and you get the audio's own boundaries, independent of your grid:
298298
| C ABI | `crispasr_diarize_segments_turns_abi(...)` (0.8.30+) |
299299
| Rust | `crispasr::diarize_segments_with_turns(...) -> Result<Vec<DiarizeTurn>, String>` |
300300
| Go | `whisper.DiarizeSegmentsWithTurns(...) ([]DiarizeTurn, error)` |
301+
| Python | `diarize_segments_with_turns(...) -> (bool, list[DiarizeTurn])` |
302+
| Dart | `diarizeSegments(..., outTurns: turns)` |
303+
| Java/JNA | `Lib.crispasr_diarize_segments_turns_abi(...)` |
301304

302-
All four are additive: the segments come back labelled exactly as they would
305+
All surfaces are additive: the segments come back labelled exactly as they would
303306
without the turns, and every other method reports zero turns rather than an
304307
error. Turn timestamps are on the **same absolute timeline as your segments**
305308
(the ABI adds `slice_t0_cs` back on the way out), so they compare directly.
@@ -316,12 +319,14 @@ Its turn buffer is caller-allocated: pass `out_n_turns` to learn the count,
316319
`out_turns` + `n_turns_cap` to receive them, and expect `2` when the buffer
317320
was short — the segments are still labelled, `*out_n_turns` holds the required
318321
capacity, and a retry costs a second full pass because the ABI keeps no state
319-
between calls. The Rust and Go wrappers size the buffer from the audio length
322+
between calls. The Rust, Go, Python, and Dart wrappers size the buffer from
323+
the audio length
320324
(one slot per 0.5 s, above FoxNose's 0.6 s embedding hop) and retry once on
321325
`2`, so their callers never see any of that.
322326

323-
Java, JavaScript and Ruby wrap `crispasr_diarize_segments_abi` but not yet the
324-
turns symbol; nothing is broken there, they simply cannot reach the turns.
327+
Java exposes both calls at its low-level JNA surface. JavaScript and Ruby do
328+
not currently expose standalone diarization; their unused native declarations
329+
are not public binding APIs.
325330

326331
## pyannote segmentation: chunked inference and the powerset layout (#326)
327332

flutter/crispasr/lib/src/crispasr.dart

Lines changed: 79 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -217,6 +217,17 @@ class DiarizeSegment {
217217
DiarizeSegment({required this.t0, required this.t1, this.speaker = -1});
218218
}
219219

220+
/// One audio-derived speaker turn returned by FoxNose.
221+
///
222+
/// Times are seconds on the same absolute timeline as [DiarizeSegment].
223+
class DiarizeTurn {
224+
final double t0;
225+
final double t1;
226+
final int speaker;
227+
const DiarizeTurn(
228+
{required this.t0, required this.t1, required this.speaker});
229+
}
230+
220231
enum DiarizeMethod {
221232
/// Stereo only. |L| vs |R| energy per segment, 1.1× margin.
222233
energy,
@@ -258,6 +269,10 @@ enum DiarizeMethod {
258269
/// WeSpeaker embedder GGUF; [minSpeakers] / [maxSpeakers] bound the
259270
/// automatic speaker-count estimation (0 keeps the library defaults 1 / 8)
260271
/// and [numSpeakers] > 0 pins the count and skips estimation.
272+
///
273+
/// When [outTurns] is supplied, the function calls the CrispASR 0.8.30+
274+
/// turn-returning ABI and replaces that list with FoxNose's audio-derived
275+
/// turns. Other methods leave it empty.
261276
bool diarizeSegments({
262277
required List<DiarizeSegment> segs,
263278
required Float32List left,
@@ -271,8 +286,10 @@ bool diarizeSegments({
271286
int minSpeakers = 0,
272287
int maxSpeakers = 0,
273288
int numSpeakers = 0,
289+
List<DiarizeTurn>? outTurns,
274290
DynamicLibrary? lib,
275291
}) {
292+
outTurns?.clear();
276293
if (segs.isEmpty || left.isEmpty) return true;
277294
lib ??= DynamicLibrary.open(CrispASR.defaultLibName());
278295

@@ -329,13 +346,68 @@ bool diarizeSegments({
329346
(optsPtr + 40).cast<Int32>().value = numSpeakers;
330347
(optsPtr + 44).cast<Int32>().value = 0;
331348

332-
final fn = lib.lookupFunction<
333-
Int32 Function(Pointer<Float>, Pointer<Float>, Int32, Int32,
334-
Pointer<Uint8>, Int32, Pointer<Uint8>),
335-
int Function(Pointer<Float>, Pointer<Float>, int, int, Pointer<Uint8>,
336-
int, Pointer<Uint8>)>('crispasr_diarize_segments_abi');
337-
final rc =
338-
fn(leftPtr, rightPtr, n, isStereo ? 1 : 0, segsPtr, segs.length, optsPtr);
349+
var rc = 0;
350+
if (outTurns == null) {
351+
final fn = lib.lookupFunction<
352+
Int32 Function(Pointer<Float>, Pointer<Float>, Int32, Int32,
353+
Pointer<Uint8>, Int32, Pointer<Uint8>),
354+
int Function(Pointer<Float>, Pointer<Float>, int, int, Pointer<Uint8>,
355+
int, Pointer<Uint8>)>('crispasr_diarize_segments_abi');
356+
rc = fn(
357+
leftPtr, rightPtr, n, isStereo ? 1 : 0, segsPtr, segs.length, optsPtr);
358+
} else {
359+
final fn = lib.lookupFunction<
360+
Int32 Function(
361+
Pointer<Float>,
362+
Pointer<Float>,
363+
Int32,
364+
Int32,
365+
Pointer<Uint8>,
366+
Int32,
367+
Pointer<Uint8>,
368+
Pointer<Uint8>,
369+
Int32,
370+
Pointer<Int32>),
371+
int Function(
372+
Pointer<Float>,
373+
Pointer<Float>,
374+
int,
375+
int,
376+
Pointer<Uint8>,
377+
int,
378+
Pointer<Uint8>,
379+
Pointer<Uint8>,
380+
int,
381+
Pointer<Int32>)>('crispasr_diarize_segments_turns_abi');
382+
var turnCap = ((n + 7999) ~/ 8000) + segs.length + 16;
383+
for (var attempt = 0; attempt < 2; attempt++) {
384+
final turnsPtr = calloc<Uint8>(turnCap * 24);
385+
final nTurnsPtr = calloc<Int32>();
386+
rc = fn(leftPtr, rightPtr, n, isStereo ? 1 : 0, segsPtr, segs.length,
387+
optsPtr, turnsPtr, turnCap, nTurnsPtr);
388+
final required = nTurnsPtr.value;
389+
if (rc == 2 && attempt == 0 && required > turnCap) {
390+
calloc.free(turnsPtr);
391+
calloc.free(nTurnsPtr);
392+
turnCap = required;
393+
continue;
394+
}
395+
if (rc == 0) {
396+
final count = required < turnCap ? required : turnCap;
397+
for (var i = 0; i < count; i++) {
398+
final base = turnsPtr + i * 24;
399+
outTurns.add(DiarizeTurn(
400+
t0: base.cast<Int64>().value / 100.0,
401+
t1: (base + 8).cast<Int64>().value / 100.0,
402+
speaker: (base + 16).cast<Int32>().value,
403+
));
404+
}
405+
}
406+
calloc.free(turnsPtr);
407+
calloc.free(nTurnsPtr);
408+
break;
409+
}
410+
}
339411

340412
if (rc == 0) {
341413
for (var i = 0; i < segs.length; i++) {

flutter/crispasr/test/bindings_smoke_test.dart

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,10 @@
77

88
import 'dart:ffi';
99
import 'dart:io';
10+
import 'dart:typed_data';
1011

11-
import 'package:crispasr/crispasr.dart' show DiarizeMethod, LidMethod;
12+
import 'package:crispasr/crispasr.dart'
13+
show DiarizeMethod, DiarizeSegment, DiarizeTurn, LidMethod, diarizeSegments;
1214
import 'package:ffi/ffi.dart';
1315
import 'package:test/test.dart';
1416

@@ -300,6 +302,7 @@ void main() {
300302
'crispasr_params_set_max_tokens',
301303
'crispasr_text_detect_language',
302304
'crispasr_enhance_audio_rnnoise',
305+
'crispasr_diarize_segments_turns_abi',
303306
]) {
304307
expect(() => lib.lookup(s), returnsNormally,
305308
reason: 'missing C-ABI symbol: $s');
@@ -339,4 +342,23 @@ void main() {
339342
reason: 'extending DiarizeMethod without bumping the C-side enum '
340343
'will silently drop the new variant');
341344
});
345+
346+
test('turn-returning diarize ABI labels segments and reports no energy turns',
347+
() {
348+
final segs = [DiarizeSegment(t0: 0, t1: 1)];
349+
final turns = <DiarizeTurn>[];
350+
final ok = diarizeSegments(
351+
segs: segs,
352+
left: Float32List.fromList(List.filled(16000, 0.5)),
353+
right: Float32List(16000),
354+
isStereo: true,
355+
method: DiarizeMethod.energy,
356+
outTurns: turns,
357+
lib: lib,
358+
);
359+
expect(ok, isTrue);
360+
expect(segs.single.speaker, 0);
361+
expect(turns, isEmpty,
362+
reason: 'only FoxNose derives audio-level speaker turns');
363+
});
342364
}

python/README.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,9 @@ for seg in s.transcribe_pcm(pcm_f32, sample_rate=16000):
6565
- `Session` — unified API across all backends compiled into `libcrispasr`
6666
- `ChatSession` — text → text chat over a GGUF chat model: one-shot and streaming generation, prompt-token counting, and cancellation through an abort predicate
6767
- `align_words(...)` — word-level CTC alignment
68-
- `diarize_segments(...)` — speaker diarization (energy / xcorr / vad-turns / pyannote)
68+
- `diarize_segments(...)` — speaker diarization (energy / xcorr / vad-turns / pyannote / FoxNose)
69+
- `diarize_segments_with_turns(...)` — label segments and return FoxNose's
70+
finer audio-derived speaker turns
6971
- `SpeakerEmbedder(spec)` — pluggable embedder ("auto"/"titanet", "indextts"/"ecapa", or a `.gguf` path)
7072
- `PyannoteCache(pcm, model)` — pre-computed pyannote-seg posteriors for cross-slice consistency
7173
- `agglomerative_cluster(embeddings, ...)` — single-linkage cosine clustering for globally stable speaker IDs

python/crispasr/__init__.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
CrispASR,
99
DiarizeMethod,
1010
DiarizeSegment,
11+
DiarizeTurn,
1112
KokoroResolved,
1213
LidMethod,
1314
LidResult,
@@ -28,6 +29,7 @@
2829
cache_ensure_file,
2930
detect_language_pcm,
3031
diarize_segments,
32+
diarize_segments_with_turns,
3133
kokoro_resolve_for_lang,
3234
list_known_models,
3335
mic_default_device_name,
@@ -47,6 +49,7 @@
4749
"CrispASR",
4850
"DiarizeMethod",
4951
"DiarizeSegment",
52+
"DiarizeTurn",
5053
"KokoroResolved",
5154
"LidMethod",
5255
"LidResult",
@@ -67,6 +70,7 @@
6770
"cache_ensure_file",
6871
"detect_language_pcm",
6972
"diarize_segments",
73+
"diarize_segments_with_turns",
7074
"kokoro_resolve_for_lang",
7175
"list_known_models",
7276
"mic_default_device_name",

0 commit comments

Comments
 (0)