feat(voiceprint): live owner recognition v1#123
Draft
rich7420 wants to merge 56 commits into
Draft
Conversation
(cherry picked from commit e3b2750)
Completes the server-side half of voiceprint: the ~10k-line pipeline was built and green but had no actual embedding model — score_turns spawned a configurable sidecar command that did not exist. This adds services/voiceprint (Python) implementing the existing stdio sidecar protocol. Two pluggable backends (VOICEPRINT_BACKEND): 'onnx' wraps sherpa-onnx speaker embedding for a CAM++ model (path via env, lazy import, clear error when the model or sherpa-onnx is absent) and deliberately lets sherpa-onnx own the matched fbank front-end to avoid front-end-parity drift; 'reference' is a deterministic, dependency-free 192-dim backend used ONLY to make the protocol and pipeline testable without weights or network — it is non-discriminative and labeled as such. The WAV reader is a direct RIFF parser (no audioop, which is gone in Python 3.13+) and matches wav.ts sample-for-sample (8-bit unsigned, 16/24/32-bit PCM, float, clip, linear resample). TS side: config.voiceprint.live_scoring can already point at any sidecar; added a dev-only dev_reference_backend opt-in that defaults the command to the bundled reference script. Live scoring stays OFF by default and voiceprintRealtimeEnabled is untouched. Extended VoiceprintModelInfo.provider (and the expected_model whitelist) with 'sherpa-onnx'/'reference' so the sidecar's model tags are first-class and expected_model enforcement works. Validation: python3 services/voiceprint/test_embed.py 26/26; voiceprint suite 177 pass / 0 fail (was 172, +5); bunx tsc --noEmit clean; CLI + TS-integration end-to-end through runEmbeddingSidecar produce valid id-matched 192-dim embeddings and a resolved score_turns round-trip. On-device (iPhone) live scoring and a real CAM++ model download are follow-ups. (cherry picked from commit edc6ef1)
…imination) Adds a dual-track end-to-end test for the voiceprint pipeline plus a mic smoke tool. Track 1 (tests/e2e-voiceprint-pipeline.ts): a deterministic, weights-free gateway e2e that drives the real identity.voiceprint.* RPC handlers and the real Python reference sidecar (no mocks) through the whole flow — register audio artifact, build turn state via realtime events, enroll the owner, score turns (self-match resolves, unrelated audio stays unresolved), apply_bundle persistence + expected_model enforcement, and realtime_reset. Runs in the repo e2e glob; always green in CI. Track 2 (tests/e2e-voiceprint-onnx.ts): the same harness with a real 3D-Speaker CAM++ model via sherpa-onnx, proving actual speaker discrimination through the gateway. Skips cleanly (never fails) when the model, sherpa-onnx, or labeled audio are absent. scripts/setup-voiceprint-model.sh provisions a venv + sherpa-onnx + the CAM++ model + labeled speaker clips; the model, audio, and venv are gitignored so nothing large enters git. Verified running for real: same-speaker cosine ~0.60 resolves, different-speaker ~0.05 stays unresolved. scripts/voiceprint-mic-smoke.sh records real voices from the mic (or scores existing clips) and prints the accept/reject matrix against a real CAM++ model, with empirically observed operating points documented in the header (owner-self ~0.88, a different real person ~0.38, a TTS voice ~0.10 on English mic audio) as threshold-calibration guidance — calibrate on real-human impostors, not TTS. tests/e2e-gateway.ts: swallow the rejection of intentionally-abandoned sendRequest calls so their 10s timeout can't leak a cross-file unhandled rejection into the alphabetically-later e2e-voiceprint specs under the single-concurrency e2e run. Validation: e2e-voiceprint-pipeline 3 pass; e2e-voiceprint-onnx 2 pass (real model); voiceprint suite 177 pass; tsc clean. (cherry picked from commit 4a0a309)
Turn scoring collapsed the enrolled owner embeddings to a mean centroid, which blurs across recording conditions: an owner recorded on a different mic/room/day drifts away from the centroid and can fall below the accept threshold even though it closely matches one enrolled clip. First real-voice measurements showed the owner self-matching at ~0.88 within a session but only ~0.60 across a different recording, close to the ~0.38 a different real person scores. Add ownerSimilarity(): the max cosine over the per-clip enrolled embeddings (the best-matching enrolled clip), skipping unusable vectors. scoreVoiceprintTurn and the offline threshold report both use it, so calibration reflects production scoring. For a single enrolled clip this is bit-for-bit identical to the old mean-of-one centroid, so single-clip behavior and its tests are unchanged; only multi-clip enrollment now benefits. This raises the genuine cross-condition score without moving the threshold (kept at 0.82/0.72), preserving the impostor margin. Demonstrated with the real CAM++ model on a dispersed enrollment set: max beats the centroid on a held-out same-speaker recording while an impostor stays well below accept. Known tradeoff (documented in the helper + the plan): max can raise an impostor's score by picking the least-far enrolled clip, so a single low-quality/outlier enrolled clip is a liability. Enrollment-quality gating, top-k mean, and AS-Norm score normalization are the planned follow-ups; this change is only the max aggregation. Validation: voiceprint suite 184 pass / 0 fail; e2e-voiceprint-pipeline 3 pass; e2e-voiceprint-onnx 2 pass (real model); tsc clean. (cherry picked from commit 3473170)
Protocol extension so the gateway can score a voiceprint turn against an embedding computed by the client (on the phone), instead of only from a server-side audio artifact via the sidecar. This is the prerequisite for on-device live scoring: the phone can send the embedding vector rather than shipping biometric audio to the server. score_turns turns now accept an optional sampleEmbedding + sampleEmbeddingModel. When present (and opted in), the turn is scored directly through the same scoreLiveVoiceprintScoringJobResponse path the sidecar uses — producing a byte-for-byte identical state/patch — and is NOT added to the sidecar job list, so a batch of only client-embedded turns never spawns the sidecar process (proven by tests with a sidecar that exits non-zero if ever run). A mixed batch runs the sidecar solely for the audio turns. Turns without a client embedding are unchanged. Guardrails (this moves the trust boundary — the server can no longer verify the audio actually produced the vector): - Opt-in only: config voiceprint.live_scoring.accept_client_embeddings, default false. When off, a supplied vector is never trusted — the turn falls back to the audio/sidecar path or is skipped, never silently accepted. - Strict validation: finite, non-zero-norm, and dimension == owner template; any bad input is rejected with a specific reason, never a spurious accept. - Model match enforced: sameVoiceprintModel between the client vector and the owner template (missing/mismatched model rejected) — vectors from different models are not comparable. - Consent gating and thresholds unchanged; live scoring off by default. Known limitation (documented): there is no per-request liveness/attestation, so a replayed or leaked owner embedding would be accepted as owner when the opt-in is on. This is inherent to trusting a client-supplied vector; it is bounded by the authenticated session and the default-off opt-in, and device attestation / a capture-binding nonce is the follow-up before enabling it in production. Server-side only — no iOS/Swift changes. Validation: voiceprint suite 197 pass / 0 fail; e2e-voiceprint-pipeline 3 pass; e2e-voiceprint-onnx 2 pass (real model); tsc clean. (cherry picked from commit b5c327c)
Wire the existing template machinery into real gateway RPCs so an owner can be enrolled, updated, and deleted — previously enrollment was ad hoc (owner embeddings passed inline in the scoring config). - identity.voiceprint.enroll_owner: embeds each audio source via the sidecar, quality-gates each clip, requires assessVoiceprintEnrollment to pass (>= 30s of voiced speech AND per-clip quality), then buildVoiceprintTemplateArtifact and writes it AES-256-GCM encrypted to the owner-template store that score_turns resolves. Rejected assessments store nothing. Consent (capture + biometric) is enforced and cannot be self-granted. - identity.voiceprint.add_enrollment_clip: appends a clip to the existing owner template (embed, quality-gate, model-match, re-assess, re-store) — grows the multi-clip enrollment the max-over-enrolled scoring benefits from. - identity.voiceprint.delete_owner_template: tombstones and unlinks the encrypted template (idempotent; removes even a corrupt/unreadable store) so a subsequent score can no longer resolve the owner — the withdrawal primitive. Security: the enrollment minimum-speech floor is a server policy the client cannot lower (a client-supplied minSpeechMs is clamped to >= 30s); the encryption key lives in a key file under the config root (losing it means re-enrollment); the raw key and embeddings are never logged or echoed. Audio paths are confined to allowedAudioRoots. Everything stays off by default; no iOS/Swift changes. Validation: voiceprint suite 206 pass / 0 fail; e2e-voiceprint-pipeline 3 pass; new e2e-voiceprint-enrollment-onnx proves enroll_owner -> score with the real CAM++ model (same-speaker resolves, cross-speaker does not) on a >= 30s voiced enrollment clip; tsc clean. (cherry picked from commit 7edd3c6)
Per-turn scoring judged each turn in isolation, so a single borderline turn could flip the 'is the owner speaking' judgment. Add a pure, session-level evidence accumulator that folds per-turn decisions into a stabilized verdict with hysteresis and the possible_owner grey band. evidence.ts (pure, deterministic — no IO/Date.now/random; time via atMs): reduceSpeakerEvidence folds each turn's decision (+ score) into a state whose verdict is owner_present / provisional / not_owner / unknown. Flips require K CONSECUTIVE turns in BOTH directions (default K=3); possible_owner is weak evidence that resets both streaks and never hard-flips alone; a single outlier turn cannot flip or downgrade a settled verdict (symmetric for owner_present and not_owner); alternating sequences stay provisional (no flapping); a stale gap decays back toward unknown. readSpeakerEvidence exposes the verdict + a confidence. Integration is opt-in: scoreVoiceprintTurnWithEvidence layers the accumulator on top of the unchanged per-turn scoreVoiceprintTurnFromEmbedding; existing callers and classifyOwnerSimilarity semantics/thresholds are untouched and off by default. Validation: voiceprint suite 225 pass / 0 fail; e2e-voiceprint-pipeline 3 pass; tsc clean. Server-side only; no iOS. (cherry picked from commit a23f012)
Close the naive-replay hole in the client-embedding path (b5c327c): a client-supplied embedding is now only trusted when it carries a fresh, single- use, session-bound nonce the gateway issued. - New pure VoiceprintLivenessNonceStore (time injected via nowMs; no timers): issueChallenge -> {nonce (256-bit opaque), expiresAtMs}; verifyAndConsume -> ok | rejected{unknown_nonce|expired|wrong_session|already_used}. Consuming burns the nonce (replay returns already_used); TTL default 60s (config voiceprint.live_scoring.liveness_nonce_ttl_ms); store is bounded per session. - New RPC identity.voiceprint.request_embedding_challenge issues a challenge for the authenticated conn.sessionKey. - When acceptClientEmbeddings is on and a turn carries a sampleEmbedding, the gateway verifyAndConsumes the turn's nonce BEFORE trusting the vector; any nonce failure rejects the turn (never falls through to scoring). Consent, model-match, and vector validation from b5c327c are still enforced — the nonce is an additional gate, not a replacement. acceptClientEmbeddings still defaults false; the audio/sidecar path is unchanged. Honest scope: A8 provides replay resistance only. It does NOT bind the nonce to the on-device capture, so a compromised client could request a nonce then submit an arbitrary vector; true defense needs device attestation + capture-binding (the iOS half + a deeper follow-up) before accept_client_embeddings ships in production. Documented in the module header, the gate comment, and the plan. Validation: voiceprint suite 243 pass / 0 fail; e2e-voiceprint-pipeline 3 pass; voiceprint-methods gateway tests 36 pass; tsc clean. Server-side only; no iOS. (cherry picked from commit 53c2101)
Raw cosine is recording-condition dependent (measured: same owner ~0.88
same-session but ~0.60 cross-recording; a different real person ~0.38), so a
single scalar threshold cannot span conditions. Add symmetric AS-Norm, which
normalizes the owner<->test cosine against a cohort of impostor embeddings to
tighten the genuine distribution and make scores comparable across conditions.
- Pure as-norm.ts: asNormScore = 0.5*((raw-mu_e)/sd_e + (raw-mu_o)/sd_o) over
the top-N cohort cosines (default N=min(300,cohort)); zero/NaN-std guarded to
fall back to the raw score. validateVoiceprintCohort enforces finite, correct
dimension, and cohort-model == owner-model (sameVoiceprintModel).
- Opt-in and additive: config voiceprint.live_scoring.as_norm { enabled (default
false), cohort, normalized_thresholds, top_n }. When enabled with a cohort,
scoring normalizes the raw ownerSimilarity and classifies with SEPARATE
normalized thresholds (AS-Norm output is a z-score scale, not cosine, so it
cannot reuse 0.82/0.72). When off (default) or no cohort, scoring is
byte-for-byte unchanged (proven at the scoring, config, and plan-wiring layers;
the real-model onnx e2e is unchanged on the default path).
Honest gating: a production cohort needs hundreds of diverse non-owner speakers
embedded with the SAME model, and the normalized thresholds must be calibrated
on real data (ties to the threshold-calibration follow-up). AS-Norm must not be
enabled in production until a real cohort + calibrated thresholds are
provisioned; only the algorithm + opt-in wiring + a synthetic test cohort ship
here.
Validation: voiceprint suite 270 pass / 0 fail; e2e-voiceprint-pipeline 3 pass;
real CAM++ onnx e2e 3 pass (default path unchanged); tsc clean. Server-side
only; no iOS.
(cherry picked from commit b9c5b90)
Add the biometric legal lifecycle (BIPA/GDPR) around the voiceprint feature. - Consent ledger: an append-only, durable record of consent grants/withdrawals per subject (identity.voiceprint.record_consent / get_consent). enroll and score consult the persisted ledger with restrict-only semantics (it can further-restrict, never widen), inert unless enforcement is configured. - Right-to-erasure: identity.voiceprint.withdraw_consent purges EVERYTHING derived from the subject via a shared purgeVoiceprintSubject primitive — the encrypted owner template (and its AES key file is crypto-shredded so the ciphertext is unrecoverable), all derived storage states/tags/signals/ annotations, the in-memory realtime turn tracker (raw-audio pointers + speech windows), and the audio-artifact cache. After withdrawal no owner resolves and no biometric derivative remains; idempotent. The append-only consent history survives the erasure of the biometric DATA. - Retention: a configured destruction window (voiceprint.retention_days/_ms with a published schedule) + identity.voiceprint.purge_expired that destroys data past the window. - Audit trail: an append-only log of enroll/score/delete/withdraw/purge events (identity.voiceprint.get_audit_log). Every record is metadata-only by an allow-list guard (assertVoiceprintAuditRecordHasNoSecrets) enforced on write and re-validated on read — no embedding, raw audio, or key can ever enter it. All lifecycle stores default to in-memory + non-enforcing, so existing behavior and call sites are unchanged; production wiring uses file-backed stores under the config root but stays non-enforcing (enforcement is a safe follow-up). Validation: voiceprint suite 282 pass / 0 fail; e2e-voiceprint-pipeline 3 pass; voiceprint-methods gateway tests 45 pass; tsc clean. Server-side only; no iOS. (cherry picked from commit f84cc4f)
Add three model-lifecycle safety mechanisms, all additive and off by default. - Reference-backend production guard (safety-critical: the reference backend is non-discriminative and would accept every speaker as owner). Config voiceprint.live_scoring.require_discriminative_model hard-rejects, at config resolve AND at runtime, any path that would score real turns with the reference backend: dev_reference_backend, a sidecar env selecting VOICEPRINT_BACKEND=reference, a reference-tagged owner template or client embedding, AND -- closing the declared-env bypass -- any sidecar RESPONSE whose returned model is reference-tagged. Enrollment is guarded too. - Model integrity pin: config voiceprint.live_scoring.model_sha256 verifies the model file hash at startup (fail-fast on mismatch). - Model-version mismatch handling: when the scoring model differs from the stored owner-template model, scoring returns a clear needs_reenrollment instead of a silent mismatched cosine. New identity.voiceprint.reembed_owner_template re-embeds from retained enrollment audio with the current model (or returns needs_reenrollment when none is retained); both emit a reembed audit entry. Validation: voiceprint suite 304 pass / 0 fail; e2e-voiceprint-pipeline + real CAM++ onnx e2e 6 pass; tsc clean. Server-side only; no iOS. (cherry picked from commit 22a7f78)
Audit and harden every voiceprint failure path so it degrades gracefully: never crash the gateway, and never falsely accept a speaker as owner. - Partial-batch isolation (the real gap): a single garbage/wrong-dimension sidecar embedding used to throw out of the batch scorer, erroring every GOOD turn in the batch. Now an unusable sample embedding (empty/NaN/inf/zero-norm/ wrong-dim) is reclassified into a per-turn scoring_failed skip while the good turns still resolve and the RPC returns partial. Structural/security faults (owner-template, consent, model-mismatch, reference-model guard) still throw as clean typed errors. - Fail-closed invariant: any sidecar fault (spawn/exit/timeout/killed/garbage/ truncated/missing-or-duplicate id/wrong-model) becomes status:error with the affected states marked error (result undefined), never resolved, never re-thrown out of the handler; a bad cosine collapses to the invalid sentinel which classifies as unknown_speaker. Lifecycle read/write RPCs, realtime_event, and sidecar/host/storage faults now surface as sanitized typed MethodErrors instead of raw INTERNAL_ERROR, and withdraw/purge stay fail-closed even if the audit append fails after the destruction phase. New tests/test-voiceprint-fail-closed.ts drives each failure mode through the real score_turns RPC and asserts, across every case, that no state resolves and nothing crashes. (Also commits the A5 reference-guard runner test left staged.) Validation: voiceprint suite 322 pass / 0 fail; e2e-voiceprint-pipeline + real CAM++ onnx 5 pass; tsc clean. Server-side only; no iOS. (cherry picked from commit d57eec8)
Add a privacy-safe telemetry sink for scoring decisions so operators can
monitor decision drift and gather the raw material for threshold calibration,
without ever recording a biometric vector, raw audio, or the encryption key.
- Each scored turn emits a record with { decision, score (scalar), thresholdUsed,
model tags, outcome, opaque sessionRef } — the sessionRef is a one-way SHA-256
hash of the sessionKey, not raw PII. An allow-list guard
(assertVoiceprintScoreTelemetryHasNoSecrets, mirroring the A4 audit guard)
runs on every write AND read across all sinks and rejects any record whose
score is array-shaped or that carries an embedding/audio/key field. Skipped and
errored turns emit outcome-only records (no score).
- Aggregation: per-decision score histograms (clamped so AS-Norm z-scores stay
bounded) plus outcome/decision counts, read via
identity.voiceprint.get_score_telemetry.
- Sinks mirror the A4 lifecycle pattern: no-op by DEFAULT (config
voiceprint.telemetry.enabled defaults false; the scoring path builds no
telemetry when off, byte-for-byte unchanged), with an opt-in in-memory or
atomic file-backed sink under the config root.
This is the field data source the threshold-calibration follow-up consumes; a
single cosine scalar is not the biometric template, and the vector/audio/key
never appear.
Validation: voiceprint suite 332 pass / 0 fail; e2e-voiceprint-pipeline + real
CAM++ onnx 5 pass; tsc clean. Server-side only; no iOS.
(cherry picked from commit d4f7cf2)
Bridge a reviewed voiceprint owner tag into the person/memory policy path so a confirmed owner-speaking turn can (only when policy permits) contribute a memory candidate, while every weaker or unreviewed signal stays quarantined. Closes the gap where nothing outside src/identity/voiceprint consumed voiceprint signals. - New pure module memory-bridge.ts: voiceprintTurnRecordsToMemoryCandidate maps a VoiceprintTurnRecords to a MemoryCandidate via the existing candidate contract and voiceprint policy (no reimplementation). Fail-closed: only a strong (score >= ownerAccept), consented, owner_speaking tag whose signal review state is confirmed yields a promotable candidate; possible_owner, unknown_cluster, unknown_speaker, rejected/suppressed/unreviewed, consent-withheld, sub-threshold, or any thrown internal error all degrade to a quarantined candidate (quarantineReason unreviewed_identity_signal, durableMemory false). The candidate review state mirrors the source signal and is never upgraded. - No secrets: assertVoiceprintMemoryCandidateHasNoSecrets (allow-list, mirroring the A4 audit and A7 telemetry guards) runs on every produced candidate and rejects an embedding vector, raw audioPath, key, or vector-shaped score. - Opt-in, no-op by default (A4/A7 posture): exposed as the gateway RPC identity.voiceprint.bridge_memory_candidate, guarded by voiceprint.memory_bridge.enabled (default false) and wired through the composition root so the flag actually toggles it; disabled, the RPC refuses with FAILED_PRECONDITION and the default distillation / person-snapshot path is byte-for-byte unchanged (git diff of src/memory is empty). Validation: voiceprint suite 350 pass / 0 fail; e2e-voiceprint-pipeline + real CAM++ onnx 5 pass; tsc clean. Server-side only; no iOS. The review UI that operates these candidates remains follow-up work. (cherry picked from commit 9b25eb4)
Make voiceprint realtime event ingestion provider-agnostic via an adapter seam, so a non-OpenAI realtime/ASR backend can drive the turn tracker instead of falling through to unsupported_event and never scoring. - New canonical internal event vocabulary (live-realtime-canonical.ts) — the normalized speech_started / speech_stopped / transcript_completed / audio_artifact form the turn tracker already consumes. - New adapter registry (live-realtime-adapters.ts): a provider adapter is a pure (rawEvent) -> canonical | null | ignored function. Ships three real adapters — openai (a faithful byte-for-byte port of the previous inline OpenAI logic: identical field-alias lists, item_id/response_id transcript fallback, and speech-window/transcript join-resolvability), a native voice.* pass-through, and a Gemini / Vertex Live adapter (activityStart/activityEnd VAD + input/outputTranscription). Adding a provider is now a data-driven adapter. - applyLiveVoiceRealtimeEvent keeps its public signature and the LiveVoiceRealtimeEventResult shape; it gains an OPTIONAL provider hint (default auto), dispatching through the registry (OpenAI first) so existing callers are byte-for-byte unchanged. The gateway realtime_event RPC and realtime session store thread provider as an OPTIONAL param defaulting to auto — no required param added. Validation: voiceprint suite 360 pass / 0 fail (new provider-ingest test 10 pass); the existing OpenAI-shaped realtime test is unmodified and still passes; e2e-voiceprint-pipeline + real CAM++ onnx 5 pass; tsc clean. Server-side only; no iOS. (cherry picked from commit 081642d)
Add the threshold-calibration machinery the scoring path needs to move off the
provisional 0.82/0.72 defaults, verified on synthetic and real CAM++ fixture
distributions. Ships the machinery plus a provisional per-model profile; final
production numbers still require a real human impostor cohort (device testing).
- New calibration.ts: computeVoiceprintOperatingPoint(genuine, impostor) computes
FAR/FRR across candidate thresholds, the equal-error-rate and its threshold, and a
recommended { ownerAccept, ownerPossible } by a stated rule (accept = lowest
threshold meeting target FAR; possible = highest candidate strictly below accept
meeting target FRR, so the possible_owner band stays non-empty on separable data).
Degenerate input (empty/one-sided/non-finite, or an all-non-finite candidate set)
returns a typed insufficient_data / no_valid_candidates result instead of emitting a
bogus threshold or crashing; every recommendation is re-checked with
validateVoiceprintThresholds (>= 0.5, accept >= possible).
- Per-model calibration profiles keyed by model identity (provider + modelId) and
score space (raw_cosine vs asnorm_zscore). resolveVoiceprintThresholdsForModel picks
a matching profile or falls back to DEFAULT_VOICEPRINT_THRESHOLDS. A CAM++
raw-cosine profile (ownerAccept 0.55 / ownerPossible 0.45) is included and flagged
provisional: true, with a doc-comment + plan note stating production calibration
needs a diverse real human cohort embedded with this exact model (TTS ~0.10 and
same-mic fixtures are explicitly insufficient).
- Field-data path: derive genuine/impostor score arrays from the A7 telemetry
histograms so a running deployment can calibrate from real usage.
- Opt-in / default-unchanged: profiles default to empty so effective live thresholds
are identical to today; DEFAULT_VOICEPRINT_THRESHOLDS is untouched. report.ts gains
an additive operating-point summary (labeled provisional); existing report output is
unchanged.
Validation: voiceprint suite 374 pass / 0 fail (new calibration test 14 pass);
e2e-voiceprint-pipeline + real CAM++ onnx 5 pass; tsc clean; DEFAULT thresholds diff
empty. Server-side only; no iOS.
(cherry picked from commit 821cd1b)
Add the iOS on-device speaker-embedder seam so a finalized turn can be embedded
locally and sent as a client embedding (privacy: no raw audio shipped) instead of
relying on the server sidecar. Compile + unit verifiable; real embedding accuracy
needs a bundled CoreML CAM++ model and device testing.
- New ios/hawky/Voiceprint: SpeakerEmbedder protocol producing a Float vector +
model info { provider, modelId, version? }; CoreMLSpeakerEmbedder loads a bundled
CAM++ .mlmodel/.mlpackage BY NAME and reports unavailable when absent (the real
model is a device-provisioned, gitignored binary, like the server campplus.onnx);
DeterministicSpeakerEmbedder is a dev/test-only, non-discriminative reference stub
so the seam and serialization are unit-testable without the real model.
- score_turns serialization matches the server client-embedding contract exactly:
sampleEmbedding (JSON number array) + sampleEmbeddingModel object + an optional
nonce string (the A8 liveness nonce that B2 will supply).
- Off by default: a new onDeviceEmbeddingEnabled flag (default false) plus the
existing voiceprintRealtimeEnabled (default false) AND CoreML availability all gate
the path; when the model is absent or a flag is off the session degrades to the
existing marker path without crashing or blocking. Default app behavior unchanged.
Validation: xcodegen + xcodebuild (iPhone 17 Pro simulator) BUILD SUCCEEDED; hawkyTests
unit bundle 328 pass incl. the new VoiceprintEmbedderTests (deterministic stability,
too-short throws, exact server keys, CoreML-unavailable fallback). Real embedding
accuracy / 192-dim discrimination + on-device audio front-end remain device testing.
(cherry picked from commit b83710b)
Bind an A8 liveness nonce to the on-device client-embedding submission so the
gateway can accept an iOS-computed embedding without shipping raw audio. Compile +
unit verifiable; the live gateway round-trip is device testing.
- LiveGatewayBridge gains requestVoiceprintEmbeddingChallenge (calls
identity.voiceprint.request_embedding_challenge, parses { nonce, expiresAtMs }) and
sendVoiceprintScoreTurns (calls identity.voiceprint.score_turns with the B1 params).
- New VoiceprintLivenessCoordinator implements the FAIL-CLOSED nonce rule: a client
embedding is never submitted without a fresh, unexpired, single-use nonce. Because
the gateway calls verifyAndConsume once per eligible turn, each embedding-carrying
turn requests its OWN nonce and is stamped individually; a batch with 2+ embedding
turns therefore carries a distinct nonce per turn (a single batch-wide nonce would be
burned on turn #1 and rejected on turn #2). If a fresh unexpired nonce cannot be
obtained for ANY embedding turn, the whole batch's embeddings are dropped with no
submission (the realtime_event marker path already reported those turns); a nonce
within a safety margin of expiry is discarded and refetched. Never reused, never
submitted nonce-less, never crashes or blocks the session.
- On-device CoreML inference is run off the @mainactor so per-turn embedding never
blocks the live session's main actor (B1 follow-up).
- Off by default and inert at the repo default: gated behind voiceprintRealtimeEnabled
+ onDeviceEmbeddingEnabled + CoreML availability; with no bundled model the path
never runs and the marker path is byte-for-byte unchanged.
Validation: xcodegen + xcodebuild (iPhone 17 Pro simulator) BUILD SUCCEEDED; hawkyTests
unit bundle passes incl. VoiceprintLivenessBindingTests (fresh-nonce attach, single-use
across submissions, per-turn distinct nonces in a multi-embedding batch, batch
fail-closed when any turn's nonce is unavailable, expiry/safety-margin, exact server
keys). Live challenge -> score_turns round-trip + real PCM/model remain device testing.
(cherry picked from commit 9a2b75d)
Add the iOS owner-enrollment UX so a user can set up their encrypted owner voice template from the app. Compile + unit verifiable; the live mic-to-template round-trip is device testing. - OwnerEnrollmentModel (SwiftUI-independent, unit-tested): tracks recorded sources, assembles enroll_owner params with the exact server keys (sources with audioArtifactId/audioPath/both-or-neither startMs/endMs/route; consent with captureAllowed/biometricAllowed/memoryPromotionAllowed/exportAllowed), enforces the FAIL-CLOSED consent gate, and validates recorded voiced duration against a guided floor (>= ~32s, tracking the server's >= 30s VOICED floor at ~74% voiced fraction), exposing explicit states (idle/recording/needsConsent/tooShort/submitting/enrolled/ failed). - Consent gate (proven by test): enroll_owner is never submitted without explicit biometric AND capture consent; consent defaults to denied; no-consent and partial-consent submissions stay in needsConsent with zero enroll calls. - LiveGatewayBridge gains enrollVoiceprintOwner and addVoiceprintEnrollmentClip RPC helpers (identity.voiceprint.enroll_owner / add_enrollment_clip). OwnerEnrollmentRecorder captures a local WAV, requests mic permission and activates a record-capable AVAudioSession before touching the engine, validates the tap format (guarding the installTap NSException), and registers the WAV as a voiceprint audio artifact. - Thin OwnerEnrollmentView linked from Settings via an explicit NavigationLink; sentence case, no forced-uppercase. Off by default: enrolling never flips voiceprintRealtimeEnabled/onDeviceEmbeddingEnabled and changes no existing screen's default behavior. Validation: xcodegen + xcodebuild (iPhone 17 Pro simulator) BUILD SUCCEEDED; hawkyTests unit bundle passes incl. 8 OwnerEnrollmentTests (consent gate, exact param/consent keys, both-or-neither ms bounds, too-short floor, enroll_owner serialization, failed-enroll handling). Real mic capture -> audio_artifact.register -> enroll_owner -> encrypted template against a live gateway remains device testing. (cherry picked from commit 5502812)
Deliver the recorded owner-enrollment WAV to the gateway so audio_artifact.register and enroll_owner can resolve it — the missing half that made on-device enrollment inert (the recorder captured locally but nothing uploaded, so the gateway never had the audio). Found and fixed via real-device integration testing. - LiveGatewayBridge.uploadVoiceprintEnrollmentAudio uploads the finalized WAV to the gateway as media.chunk.upload chunks under the same media id it then registers, so the WAV lands in an allowed audio root and the existing register + enroll_owner calls resolve it. The recorder now uploads-then-registers and only returns the artifact-backed source when both succeed, falling back to the local-path source otherwise (behavior unchanged on failure). Exposed via a protocol requirement + default no-op extension so test/inert gateways stay green. - Refactor (no behavior change): the enrollment upload and the existing deferred recording upload now share one LiveMediaAudioChunkUploader.uploadWavAsMediaChunks + one pcm16Data, instead of a duplicated chunk-upload loop and three copies of the PCM16 conversion. DeferredLiveMediaUploader is byte-for-byte preserved (same media id, sampleRate*10 chunking, captured_at_ns, skip-empty/final-on-empty, wire params). - Cleanup: drop 3 redundant awaits on synchronous calls in LiveSessionStore; use @preconcurrency import CoreML so the MLModel-backed embedder is Sendable-clean; make OwnerEnrollmentModel.guidedVoicedFloorMs a nonisolated static let. Build is now warning-free for these files. Validation: xcodegen + xcodebuild (iPhone 17 Pro simulator) BUILD SUCCEEDED, 0 warnings; hawkyTests full bundle passes (OwnerEnrollment / VoiceprintEmbedder / VoiceprintLivenessBinding included). Server enroll->template->score chain independently proven against the real config + real CAM++ model (owner cosine ~0.65 resolves, different speaker ~0.045 rejected). Real device build requires the operator's Xcode signing. Server-side unchanged; no *.ts touched. (cherry picked from commit e2cc910)
Behavior-preserving DRY cleanup from the voiceprint code-quality audit; no runtime change (voiceprint suite 374 pass, unchanged; tsc clean). - Extract one normalizeCosineSimilarityToConfidence into similarity.ts; turn-scoring (confidenceFromCosineSimilarity) and evidence (normalizeScore) now delegate to it. Both originals were bit-identical (non-finite -> 0, else clamp (x+1)/2 to [0,1]). - Centralize the voiceprint provider allow-list as one top-level VOICEPRINT_PROVIDERS const (as const satisfies the provider union so member drift is compile-checked); the three previously-inlined identical lists in voiceprint-methods.ts (as-norm cohort model, client-embedding model parse, configured-model resolve) now reference it. Deliberately NOT merged: the two AS-Norm threshold validators in calibration.ts (clampRecommendedThresholds silently clamps/floors; resolveVoiceprintThresholdsForModel validates-and-throws) have genuinely different semantics and error text, so folding them would change behavior — left separate per correctness-over-DRY. Validation: bunx tsc --noEmit clean; bun test voiceprint suite 374 pass / 0 fail. Server-side only. (cherry picked from commit ded4b0f)
Behavior-preserving DRY cleanup from the code-quality audit; no runtime change (voiceprint suite 374 pass, unchanged; tsc clean). - New live-validators.ts with validateTimeBounds, validateIdentifierNotEmpty, firstDuplicate, checkNoDuplicateIds, and validateIsoLikeTime. Each carries a label/message parameter so every call site's exact error string is preserved byte-for-byte. - Replaced the inline copies: turn/identifier/time checks in live-adapter, live-sidecar-jobs, and contracts (validateSpeechTurn); the duplicate-id guard and firstDuplicate helper in live-sidecar-runner; and the ISO-time validation shared by template and template-store (keeping the distinct 'template' vs 'template file' wording via the label param). Deliberately NOT merged (genuinely different semantics, not duplicates): live-turn- tracker validateFiniteTime (single-value, non-negative), and the sidecar-protocol embedding-request time / duplicate-id checks (per-field non-negative operators and distinct request/response messaging). Validation: bunx tsc --noEmit clean; bun test voiceprint suite 374 pass / 0 fail. Server-side only. (cherry picked from commit 15ed9bf)
Readability cleanup from the code-quality audit; values and behavior byte-identical (voiceprint suite 374 pass, iOS build+tests pass, embed.py parses, tsc clean). - Name previously-inline magic numbers with their meaning, values unchanged: audio-features (MIN_ANALYSIS_SAMPLES 160 / MIN_ANALYSIS_SECONDS 0.05; the Goertzel voice-band probe frequencies [120,220,440,880,1760,3200], now mapped instead of six literal calls), wav (BITS_PER_BYTE 8; PCM 8/16/24/32 half-scale divisors), OwnerEnrollmentRecorder (tapBufferFrames 4096), embed.py (SPEECH_ABSOLUTE_RMS_FLOOR 1e-4 / SPEECH_RELATIVE_RMS_FRACTION 0.35; reference-backend BIAS_SEED 0xBEEF / BIAS_SCALE 1e-3). - Hoist the CAM++ embedding dimension (192) to one SpeakerEmbedding.camPlusDimension referenced by both embedders; the intentional reference-vs-cam++ provider/modelId distinction is preserved (custom/campplus-coreml vs reference/reference-hash-v1, not merged). Extract one setOptionalString helper for the repeated if-let/non-empty dictionary building. Comment trim (audit #6) intentionally made no deletions: every comment in the reviewed files is protected rationale (fail-closed, consent, no-secrets, honest limitations, device-testing notes) — kept verbatim rather than risk stripping intent. Validation: tsc clean; voiceprint suite 374 pass; iOS BUILD + hawkyTests SUCCEEDED (0 new warnings); embed.py parses; all named values byte-identical. (cherry picked from commit d3ff4d1)
Behavior-preserving efficiency/readability cleanup from the code-quality audit; no output change (voiceprint suite 374 pass; embed.py output proven byte-identical to HEAD across all fixture WAVs; tsc clean). - report.ts summarizeRows: fold six separate rows.filter(...).length passes into one reduce building all per-decision buckets in a single pass; identical predicates and counts. - consent-ledger.ts: extract the all-false empty-scopes literal into one emptyConsentScopes() factory used at both the initial-effective-state and withdrawal-reset sites (each returns a fresh object, so the prior non-aliasing behavior is preserved). - embed.py: read VOICEPRINT_MODEL once at OnnxBackend construction instead of twice; hoist the RIFF parser's payload_end clamp into the data-chunk branch where the span is actually used. Reference-backend embed on the fixture WAVs yields byte-identical embeddings / speechMs / rms before and after. Not done (out of scope): the loadConsentLedger read-once optimization lives in src/gateway/voiceprint-lifecycle.ts, not the consent-ledger core — deferred to keep this change to the audited files and avoid touching the append-only ledger IO. Validation: bunx tsc --noEmit clean; voiceprint suite 374 pass / 0 fail; embed.py HEAD-vs-worktree embedding/audio/model identical (192-dim). (cherry picked from commit edc3e8b)
Replace fragile error-message substring matching with a typed error so the batch scorer's skip-a-turn vs fail-the-batch decision no longer breaks if an upstream validation message is reworded. Behavior-equivalent; adds regression tests. - live-sidecar-jobs reclassifyUnusableEmbeddingError / reclassifyUnusableSampleEmbeddingError now detect the per-turn data fault by instanceof UnusableVoiceprintEmbeddingError instead of message.includes(...). The typed error moves to a leaf module embedding-errors.ts (no import cycle) and is thrown at the two validation origins: sidecar-protocol validateEmbeddingResponse (the vector usability fault) and turn-scoring validateTurnScoreEmbeddings (the two sample-embedding faults). Message text is preserved verbatim at every throw site. - Skip-vs-fail set is identical to before: exactly the three strings the substrings matched are the three typed throws; owner-embedding / consent / quality / id-mismatch / reference-model faults stay plain Error and still hard-fail; the strict transport parser still hard-fails via the sidecar-client catch. The runner's skip catcher (instanceof UnusableVoiceprintEmbeddingError) is unchanged. - Clarifying comments (behavior unchanged, comment-only): document that actorForResult never receives possible_owner (buildEventParticipation gates on allowedUses.eventGraph, which is false for possible_owner), and that policy.ts's non-finite -> -1 confidence default is a deliberate fail-closed floor (matching INVALID_VECTOR_SIMILARITY). Validation: bunx tsc --noEmit clean; voiceprint suite 379 pass / 0 fail (+5 new typed- path tests: instanceof reclassification for empty/wrong-dimension embeddings, reword-immunity, and precondition faults still hard-failing). Server-side only. (cherry picked from commit 416649e)
Behavior-preserving function decomposition from the code-quality audit; no runtime change (voiceprint suite 379 pass; iOS build + hawkyTests pass; tsc clean). - live-plan.ts: the three non-error success return paths of runLiveVoiceprintScoringPlan built a byte-identical LiveVoiceprintScoringPlanRun; extract one buildPlanRunResult helper they all call, and hoist the shared createdAt derivation. The error path is left as-is (it carries an extra error field). - OwnerEnrollmentRecorder.swift: extract resolveTapFormat, installTapIfNeeded, and finalizeSource so start()/stop() read as sequences. Control flow is relocated verbatim — same mic-permission fail-closed, session activation, tap-format fallback, and upload-then-register-then-local-fallback behavior. Left as-is deliberately: validateClientEmbedding's eight sequential fail-closed checks map 1:1 to ClientEmbeddingRejectionReason values in a test-asserted short-circuit order; explicit sequential checks are the correct, safest form for that security- sensitive path, so it was not table-ified. Validation: bunx tsc --noEmit clean; voiceprint suite 379 pass / 0 fail; iOS BUILD SUCCEEDED (0 new warnings) + hawkyTests 25 pass. (cherry picked from commit ccb83ca)
Device testing reached enroll_owner end-to-end but it was rejected quality_rejected= clipped: the built-in-mic recording ran fully hot (measured rms ~0.4-0.5, peak pinned at 1.0, ~5-8% clipped) because the capture session used mode .default, whose input AGC boosts normal speech to full scale. Fix the level and improve the recording UX. - Level (stop the clipping): OwnerEnrollmentRecorder captures with AVAudioSession mode .measurement (raw, no input AGC) instead of .default, plus a best-effort setInputGain headroom guarded by isInputGainSettable; both in do/catch so a rejecting route still records, and the deactivate/restore path is preserved. The server quality gate is unchanged (clipping genuinely harms embeddings). - Decouple display from upload: stop() returns immediately with a local-path source and the voiced counter + consent gate update at once; upload+register run in a background task that upgrades the source to artifact-backed. Enroll is gated on per-source upload state and submit() waits out every in-flight upload before enrolling, so a still- uploading source can never reach enroll_owner; the fail-closed consent gate is intact. - Live counter: the recorder publishes elapsedMs via a main-actor timer so 'Voiced speech: Ns' climbs in real time while recording. - Re-record: a 'Start over' button clears the captured clips and returns to idle without leaving the screen (keeps the biometric-consent toggle); a late background-upload callback for a cleared clip is a safe no-op. Validation: xcodegen + xcodebuild (iPhone 17 Pro simulator) BUILD SUCCEEDED, 0 new warnings; hawkyTests pass incl. enroll-awaits-pending-upload, failed-upload-fallback, and reset/start-over (keeps consent, orphans late uploads). Real mic level requires a device retest to confirm the recording no longer clips. (cherry picked from commit f18d289)
Enrollment UX polish (presentation only; no logic/gating change). - Add a state-driven 'next step' banner under the header that always tells the user the single most-important next action (Step 1 record ~30s -> Step 2 turn on consent -> you're ready, tap Enroll), so the flow leads forward instead of leaving the user to guess. Clears once enrolled; shows 'finishing upload' while a clip uploads. - Add a grey hint under the 'Start over' button explaining it clears the clips to re-record (e.g. if unhappy with the recording). Validation: xcodebuild (iPhone 17 Pro simulator) BUILD SUCCEEDED. The fail-closed consent gate, guided voiced floor, and enroll gating are unchanged. (cherry picked from commit 6a950c0)
Wire live owner recognition on device: score finalized turns against the enrolled
owner template and surface owner/unknown in the live session. Closes the gap where the
iOS live session sent realtime markers but never scored them.
- Extend LiveVoiceprintScoreTurnsResult to parse the gateway score_turns response
states[] — per turn { transcriptItemId, lifecycle, result, confidence } (the wire key
for the scalar is confidence, not score). This carries the recognition decision back
to the app.
- Add a server-side recognition path for finalized turns, gated on
voiceprintRealtimeEnabled (default off): the live recording is already uploaded to the
gateway, so each finalized turn is submitted marker-only (no on-device embedding, no
liveness nonce) via the existing sendVoiceprintScoreTurns; the gateway sidecar scores
the audio against the owner template. It deliberately does NOT route through the
liveness coordinator (that path fails closed without a nonce, which server-side sidecar
scoring does not use).
- Score each turn exactly once: the on-device embedding path and this server-side path
are mutually exclusive per batch (any on-device embedding -> the B2 client-embedding
submission; otherwise -> server-side marker-only). In the default build (no bundled
CoreML model) every turn scores server-side.
- Surface the result as a per-turn verbose line (owner vs unknown + confidence). Scoring
runs off the main actor and is fail-safe: a nil/empty/errored result logs a neutral
line and NEVER surfaces a false owner.
Validation: xcodebuild (iPhone 17 Pro simulator) BUILD SUCCEEDED, 0 new warnings;
hawkyTests pass incl. new VoiceprintServerSideRecognitionTests (states parse for
owner_speaking/unknown_speaker, marker-only turn shape, and fail-safe never-owner cases).
Server-side proven separately: the owner scores ~0.97 (owner_speaking) and a different
speaker ~0.50 (unknown_speaker) against the enrolled template. Off by default; iOS only,
no server change. Live behavior is a device retest with voiceprintRealtimeEnabled on.
(cherry picked from commit 9a95c7f)
Expose voiceprintRealtimeEnabled as a Live-settings toggle so the owner can turn on live recognition (previously the flag had no UI and could never be enabled). - Add updateVoiceprintRealtimeEnabled on LiveSessionStore (sets the flag + persists), mirroring the existing config-toggle setters. - Add a 'Live voice recognition' toggle in the Settings > Live 'Voice identity' section, under Voice enrollment, with a caption that it recognizes the owner during a live session and to enroll first. Off by default. Validation: xcodebuild (iPhone 17 Pro simulator) BUILD SUCCEEDED. iOS only. (cherry picked from commit d113603)
Live owner recognition never ran because no turn finalized. Root cause: the OpenAI Realtime WebRTC transport delegate is arg-less, so the provider re-emits input_audio_buffer.speech_started/stopped with an EMPTY JSON body, and the voiceprint converter dropped any VAD event that lacked audio_*_ms or a recording offset (nil during parallel-mic warm-up). Only transcripts survived, so the gateway turn tracker had no speech window and stayed at finalized_turns:0 — score_turns never fired. - VAD events now survive the converter and are stamped with a RECORDING-ALIGNED, finite, strictly-monotonic timestamp from recordingSink.currentAudioOffsetMs (the WAV write position). This is the correct time base: the audio artifact IS that WAV, so a finalized turn's [startMs,endMs] maps to the right recording segment — uptime/Date() would point at the wrong audio and is deliberately not used. During warm-up (offset nil) the stamp floors to the recording origin; start/stop resolving to the same offset is nudged +1ms so endMs > startMs always. - Warm-up audio-artifact race: for WebRTC the parallel-mic WAV opens lazily on the first streamed chunk, so a first-turn speech_stopped could find currentAudioArtifact nil and permanently drop the artifact (the server then never finalized that turn). The join keys are now stashed (de-duped by item-id-first-else-speech-window-id) and flushed the instant the WAV opens, so the leading first turn binds its audio instead of being lost. Per-session state is cleared on session start/stop. - Off by default and no regression: the whole path stays gated on voiceprintRealtimeEnabled / the realtime runtime target (flag off => byte-for-byte no-op), and other speech-event consumers (barge-in playback stop, timeline) are untouched — they still receive only the real item id. - Corrected the one existing test that pinned the old drop-the-event behavior; it now asserts VAD survives with a deferred stamp. Validation: xcodebuild (iPhone 17 Pro simulator) BUILD SUCCEEDED, 0 new warnings; hawkyTests pass incl. the new VoiceprintRealtimeVadTests (empty-JSON survives, monotonic endMs>startMs, warm-up late-bound artifact join, off-by-default). iOS only, no server change. The end-to-end live finalize -> score_turns -> owner/unknown is a device retest. (cherry picked from commit b926233)
Make the gateway recognize the owner itself: when the realtime turn tracker finalizes turns, score them where the audio and tracker already live, feed the A2 evidence reducer, and push the identity to clients — instead of the phone round-tripping finalized turns back through score_turns (which skipped every turn because the live recording uploads deferred). This is WS1 of the Phase 1 live-recognition architecture documented in the plan. - Config voiceprint.live_scoring.auto_score_finalized (default false; added additively to HawkyConfig). When off, behavior is byte-for-byte unchanged. - New module voiceprint-auto-score.ts owns the background orchestration: per-session single-flight batching with a queue, wait-for-audio (3x2s retries then a fail-safe skip), an A2 evidence fold keyed by sessionKey, an EDGE-TRIGGERED voiceprint.identity broadcast on verdict establish/flip (scalars only — sessionKey/verdict/decision/ confidence/at, never embeddings/audio/keys), and a capped pendingScoredStates buffer drained onto the session's next realtime_event response (additive fields). - The score_turns handler internals were extracted behavior-preservingly into scoreVoiceprintTurnsForSession and reused by the auto-scorer, so allowed-root enforcement, storage bundles, A4 audit, and A7 telemetry come free (no duplicated pipeline). realtime_reset and right-to-erasure purge the auto-score state; late results from a superseded batch are dropped via a map-identity guard. - Fail-safe: the fire-and-forget task never rejects and never blocks or fails the realtime_event response; missing audio / scoring faults skip and never yield a false owner. Validation: bunx tsc --noEmit clean; voiceprint suite 387 pass / 0 fail (8 new WS1 tests: flag-off byte-identical response, flag-on piggyback + telemetry, retry-then-skip, exactly-one edge-triggered broadcast with a no-secrets payload assertion, reset/erasure clearing, and a scoring throw producing zero unhandled rejections); real CAM++ e2e 5 pass. Server-side only; off by default. Delivering the identity to the phone + injecting it into the live agent is WS2 (iOS). (cherry picked from commit b355336)
iOS side of live owner recognition (WS2): consume the identity the gateway now pushes, tell the answering agent who is speaking, and stop the client-side score_turns round-trip that the gateway auto-score replaces. - Remove the redundant server-side score_turns path: iOS now submits score_turns ONLY when it has a real on-device embedding (Phase 2); every marker-only finalized turn is scored by the gateway auto-scorer, so the old marker-only client submission (which double-scored) is gone. The bridge score_turns helper stays (still used by the B2 on-device liveness path). - New LiveVoiceprintIdentity state machine: consumes the identity from two channels — the additive scoredStates/identity fields piggybacked on the realtime_event response (primary) and the voiceprint.identity broadcast EventFrame (secondary) — de-duped. It is EDGE-TRIGGERED: it acts only on an identity establish/flip to a hard verdict (owner_present / not_owner), never per turn and never on a same-verdict update. An unrecognized/garbled verdict maps to unknown and NEVER to owner (fail-safe). - On an identity edge it (a) surfaces an owner/unknown indicator + relabels the turns via the existing recognition-line renderer, and (b) injects exactly one short system context item into the OpenAI Realtime session via sendContext(createResponse: false) — appended with runImmediately:false so it informs the next answer without triggering a model turn — so the answering Hawky knows the owner is speaking. Best-effort: a send failure is swallowed and can never stall the conversation. - Couple liveUpload (closes the recognition-needs-live-audio gap): when live voice recognition is on, the parallel-mic recording runs in liveUpload so the gateway has each turn's audio to auto-score in time. - All of this is gated on voiceprintRealtimeEnabled; off by default it is byte-for-byte unchanged. Validation: xcodebuild (iPhone 17 Pro simulator) BUILD SUCCEEDED, 0 new warnings; hawkyTests 390 pass incl. new VoiceprintLiveIdentityTests (edge-triggered establish/flip once, cross-channel de-dupe, garbled-never-owner, off-by-default) and the updated recognition-line rendering test. iOS only; the gateway auto-score (WS1) must be enabled for end-to-end live recognition — that plus the device run is WS3. (cherry picked from commit 7270b4e)
The WS1 gateway auto-scorer produced identity for the answering Hawky but
never fired on real device audio: the live realtime path never calls
audio_artifact.register, so finalized turns resolved to no audio and were
skipped ("audio never resolved").
- resolveLiveTurnAudioArtifact: after the artifact-store miss, resolve the
turn's audio gateway-autonomously by treating its audioArtifactId as an
on-disk media id under the allowed roots (the live path uploads segments
whose media_id == the turn's artifact id).
- Split iOS's joined <recordingBase>:<turnId> artifact id on ':'.
- Map recording-aligned turn windows onto the segmented live recording
(<base>.segNNN.mic.wav) via cumulative sidecar durations and slice the exact
overlap window; no min-length padding (padding dilutes the CAM++ embedding).
- Expose the A2 evidence hysteresis via config
(voiceprint.live_scoring.evidence.flip_threshold / window_size /
stale_timeout_ms) so owner-identity establish + broadcast speed is tunable.
- Log audio_artifact_ids + turn_windows_ms on the auto-score skip warning.
## Validation
- bunx tsc --noEmit: clean
- bun test ./tests/*voiceprint*.ts: 393 pass / 0 fail
- Offline repro on real device recordings: finalized live turns resolve and
score; a held-out session recognizes the owner (owner_speaking 0.79-0.84);
auto-scorer broadcasts owner_present at flip_threshold 2.
## Limitations / Follow-ups
1. No fixture-based regression test yet for the segmented resolver (verified
via offline repro against real recordings).
2. The owner template must be enrolled in the live capture domain; the
standalone enrollment recorder domain is orthogonal to the WebRTC live tap.
3. iOS main connection does not auto-reconnect after a gateway restart.
(cherry picked from commit 89f0ed2)
Live device testing showed the owner verdict flapping: conversational
sub-2s utterances ("mm-hm", "okay") carry too little speech for a reliable
CAM++ embedding, score below the owner threshold, and — counted as
unknown_speaker evidence with a symmetric flipThreshold — repeatedly
overturned an established owner_present to not_owner mid-conversation.
- evidence.ts: asymmetric hysteresis. Optional ownerFlipThreshold /
nonOwnerFlipThreshold override flipThreshold per direction, so establishing
the owner is fast while overturning requires sustained clear non-owner
evidence. Defaults preserve symmetric behavior.
- voiceprint-auto-score.ts: minEvidenceTurnMs tuning. An unknown_speaker
decision from a turn shorter than the floor is NEUTRAL — it neither votes
toward not_owner nor resets the owner streak (short-turn "unknown" means
"could not tell", not "someone else"). Positive decisions always vote.
Also: skip-warning log now includes audio_artifact_ids + turn windows.
- voiceprint-methods.ts / types.ts: config surface
voiceprint.live_scoring.evidence.{flip_threshold, owner_flip_threshold,
non_owner_flip_threshold, window_size, stale_timeout_ms, min_turn_ms}.
## Validation
- bunx tsc --noEmit: clean
- bun test tests/*voiceprint*: 401 pass / 0 fail (8 new tests: asymmetric
establish/overturn/streak-reset, short-turn neutrality incl. default-off)
- Device acceptance: with owner_flip_threshold 2 / non_owner_flip_threshold 4 /
min_turn_ms 2000 the verdict stayed owner_present for a full mixed
long/short-utterance session (previously flipped to not_owner repeatedly).
## Limitations / Follow-ups
1. Evidence counts turns; a duration-weighted or rolling-embedding aggregate
(scoring the average of recent turn embeddings) is the deeper fix for
short-utterance identification.
2. min_turn_ms uses the turn window duration, not the sidecar's measured
voiced ms.
(cherry picked from commit 5d1b6f7)
With the plain injection ("the current speaker has been identified as the
device owner") gpt-realtime-2 still answered "I can't recognize you by your
voice": OpenAI realtime models are trained to disclaim voice recognition and
that prior beats a bare context fact. A/B smoke tests against the real model
found two requirements: attribute the verification to Hawky's voiceprint
system (the app verified it, not the model), and tell the model to respond
like a familiar person instead of reciting the mechanism.
New injection text (all three verdicts): the model confirms identity briefly
and warmly ("Yep — I know I'm talking to you, the owner"), explains the
voiceprint verification only when asked HOW it knows, and never announces a
guest/unverified transition unprompted.
## Validation
- A/B on gpt-realtime-2 (real API): old text -> refusal; new text -> confident
confirmation; mechanism explained only on follow-up; no identity chatter on
unrelated questions.
- xcodebuild test -only-testing:hawkyTests/VoiceprintLiveIdentityTests: 12
pass / 0 fail; device build clean.
- Device acceptance: full loop confirmed on iPhone (recognize -> broadcast ->
inject -> model addresses the owner naturally).
## Limitations / Follow-ups
1. The injection says "device owner", not the owner's name; the owner->name
link comes from memory/persona.
2. Wording is tuned against gpt-realtime-2; other realtime backends may need
their own phrasing pass.
(cherry picked from commit 9888c97)
Post-merge review of the live-recognition commits surfaced one latent bug and several edge risks; all gateway-side, all covered by new regression tests. - Whole-file tier-2 resolution now passes the turn window EXPLICITLY and only when the file's sidecar duration covers it: leaving requestStartMs/EndMs undefined let the scoring queue's turn.startMs fallback re-slice an arbitrary equally-named file with recording offsets (empty audio). - Segmented resolution dedupes the same physical segment reached via nested/overlapping allowed roots (by path/realpath) instead of declaring it ambiguous and permanently dropping every turn. - 250ms drift tolerance between phone frame-counter turn stamps and sidecar durations, so a session's final turn still resolves; anything further past the recorded audio stays fail-safe skipped. - Short-turn evidence skips still refresh the staleness clock: a stretch of only sub-floor turns no longer decays a settled owner to unknown. - Evidence config typos (non-integer thresholds, flip > window) now fail fast at config load instead of silently killing every scoring batch at fold time. - Fixed a false doc invariant (possible_owner DOES reset streaks). ## Validation - bunx tsc --noEmit: clean - bun test tests/*voiceprint*: 404 pass / 0 fail (3 new regression tests: mid-recording segment mapping, tail-drift tolerance both sides, overlapping -root dedupe — all through the real score_turns seam + reference sidecar) ## Limitations / Follow-ups 1. Tier-2 resolution is not yet bound to the uploading session (tracked: cross-session artifact reference on a multi-user gateway). 2. Drift tolerance is a fixed 250ms, not config surface. (cherry picked from commit 26d017c)
Documents the shipped live owner-recognition stack end to end: dataflow (phone mic -> segmented upload -> gateway auto-score -> CAM++ sidecar -> evidence -> identity push -> LLM injection), the two-tier audio resolution, owner-sticky evidence semantics, injection-wording rationale, component map, config surface, security/privacy properties (incl. the tracked tier-2 session -binding gap), lessons from device acceptance (capture-domain mismatch, timeline clocks, unknown-vs-inconclusive, attribution prompting, padding), and a three-layer benchmark proposal (model FAR/FRR, evidence simulation, tau-Voice style end-to-end multi-speaker scenarios). ## Validation - Content cross-checked against the code inventory (modules, RPC surface, config schema) and this week's device-acceptance measurements. ## Limitations / Follow-ups 1. Benchmark proposal is a plan, not an implementation. 2. Phase 2 (on-device embeddings) is referenced but not yet designed in doc form. (cherry picked from commit 352b1c1)
Root cause of the owner never matching at recognition time: enrollment captured RAW audio (a bespoke AVAudioEngine tap in .measurement mode) while live recognition scores audio from MicAudioSource(voiceProcessing: true) — Apple's voice-processing I/O (AEC/AGC/NS). For the SAME speaker those two acoustic domains are near-orthogonal to CAM++ (measured cosine 0.01-0.14 cross-domain vs ~0.6-0.7 in-domain), so a .measurement-enrolled template is useless for live scoring. This was worked around once by manually re-enrolling from live segments; this fixes it at the source. OwnerEnrollmentRecorder now captures through the EXACT same path recognition uses: MicAudioSource(enableVoiceProcessing: true) feeding LiveRecordingSink, under the .voiceChat voice-capture session (matching LiveAudioSampleRecorder). The bespoke tap, .measurement session, and manual pcm16 conversion are gone. Voice-processing AGC also keeps levels off the clipping ceiling (the reason .measurement was originally adopted) without disabling the processing. ## Validation - xcodebuild build (device): clean compile (signing-only failure). - xcodebuild test: OwnerEnrollmentModelTests + VoiceprintLiveIdentityTests pass (the recorder itself is device-only, not simulator-exercised). - Manual re-enrollment from this exact source (live segments) previously produced a template that recognized the owner at 0.6-0.7 in-domain. ## Limitations / Follow-ups 1. Standalone enrollment approximates the live domain via .voiceChat + voice processing; it is not literally inside a WebRTC session, but the dominant domain factor (the voice-processing I/O unit) is identical. 2. Requires a device re-enroll to replace any .measurement-era template. (cherry picked from commit 719fc3a)
The enrollment screen always presented a blank first-time flow even when an owner template already existed. Adds a status query so the UI can show "already enrolled" + when / how much speech / quality, framing a re-record as a replacement. - Gateway: identity.voiceprint.owner_template_status returns SCALAR metadata from the template header (enrolledAt, speechMs, sourceCount, quality, embeddingDim, model) — never the centroid/embeddings (A7). Reports enrolled:false (never throws) when enrollment is unconfigured, the file is missing, deleted, or unreadable, so the UI degrades to the first-time flow. - iOS: LiveGatewayBridge.fetchOwnerTemplateStatus + LiveVoiceprintOwnerTemplateStatus parser; OwnerEnrollmentModel.loadEnrollmentStatus (best-effort, protocol default nil for inert/test gateways); OwnerEnrollmentView queries on appear and renders an "already enrolled" banner with an enrolled-date/seconds/quality summary and a "recording replaces your template" note. ## Validation - bunx tsc --noEmit: clean - bun test tests/*voiceprint*: 406 pass / 0 fail (2 new: status before/after enroll + after delete with no biometric leak; not-enrolled when unconfigured) - xcodebuild build (device): clean compile; iOS enrollment + identity suites pass ## Limitations / Follow-ups 1. The banner does not auto-refresh after an in-screen enroll completes (the state machine already shows the enrolled result); it refreshes on next appear. (cherry picked from commit 4573304)
Pure code motion: voiceprint-methods.ts (5196 lines) splits into voiceprint-config.ts (config resolution + AS-Norm + model guards, 750), voiceprint-audio-resolve.ts (segmented/media/path resolution, 448), voiceprint-enrollment.ts (embed sources + template store I/O, 312), voiceprint-param-utils.ts (leaf validators + errorMessage, 177), leaving voiceprint-methods.ts (3647) as the RPC registration + scoring composition root. Dependency direction is one-way (methods -> helpers); back-references are import-type-only, so no runtime cycles. All 18 public exports remain importable from voiceprint-methods.ts under identical names (3 via re-export); none of the 13 importer files changed. Three previously-internal enrollment interfaces became exported (additive) for the enrollment module. ## Validation - bunx tsc --noEmit: clean - bun test tests/*voiceprint*: 406 pass / 0 fail (identical to pre-refactor) - git diff: importers untouched; behavior byte-for-byte preserved by review ## Limitations / Follow-ups 1. Score-plan builder, storage snapshot I/O, lifecycle, telemetry, and param parsers stay in voiceprint-methods.ts — tightly bound to the registration closure; further splitting would force many cross-exports for no cohesion gain. (cherry picked from commit 82c1b0d)
Gateway half of capture-domain-parity enrollment (option A). The standalone
recorder cannot engage the voice-processing unit WebRTC runs (measured: raw
capture clips at 6.5% / rms 0.48 while live segments sit at rms 0.05, and raw
audio is acoustically orthogonal to the recognition domain), so the template
must be built FROM live-captured audio itself.
- identity.voiceprint.enroll_owner_from_recording {recordingBaseId, consent,
minSpeechMs?}: selects the recording's finalized .segNNN.mic segments
(timeline order, quality-DROP for silence/echo segments, ~90s cap), then
runs the SAME enrollment seam enroll_owner uses.
- runOwnerEnrollment: enroll_owner's embed->assess->store flow extracted so
both entry points share quality gates, the voiced floor, audit posture, and
biometric-minimized responses (verified byte-identical for enroll_owner).
- collectFinalizedVoiceprintSegments: segment listing extracted from the turn
resolver (verbatim move) and shared.
- Response carries honest reconciling counts (considered = used + rejected +
capped + afterGap) so the client can render "keep talking N more seconds"
instead of a bare rejection; a broken segment timeline reports afterGap /
no_usable_segments, never a fake quality_rejected.
## Validation
- bunx tsc --noEmit: clean
- bun test tests/*voiceprint*: 412 pass / 0 fail (6 new: happy path with
quality-drop + score-resolution, not-enough-speech shape, all-rejected
shape, unknown-recording/consent/bad-id errors, 90s cap counts, gap counts)
- Post-implementation review pass fixed: double error-audit on the recording
path, gap-vs-quality mislabeling, response-shape inconsistency.
## Limitations / Follow-ups
1. iOS half pending: enrollment screen driving a silent live listening
session and calling this RPC with the recording base id.
2. Selected WAVs are read+quality-assessed twice (selection + embed seam);
correct but redundant I/O, bounded by the 90s cap.
3. recordingBaseId is not yet bound to the requesting session (same accepted
gap as tier-2 turn resolution).
(cherry picked from commit f96c6a1)
iOS half of capture-domain-parity enrollment (gateway half: f96c6a1). The enrollment screen now runs a SILENT live session through the SAME WebRTC pipeline recognition scores — a standalone recorder cannot engage that voice-processing domain (raw capture clips at 6.5% and is acoustically orthogonal to the recognition domain) — and the gateway builds the template from the recording's uploaded segments. - LiveSessionStore: startEnrollmentListeningSession() reuses the one true start()/stop() machinery with a NON-PERSISTED config override (mic on, liveUpload, camera off, assistant silenced); currentRecordingBaseId exposes the uploaded segments' base id. Silencing uses Stay Silent's setSilenceMode(true) — NOT safetyCheck/hardQuiet, which keeps VAD auto-response on and would answer the user's enrollment monologue. - OwnerEnrollmentModel: listening flow (startListening / stopListening / submitFromRecording) behind the same fail-closed biometric-consent gate; server not_enough_speech maps to an actionable "keep talking ~N more seconds" using the server-counted speechMs vs the 30s floor. - OwnerEnrollmentView: start/stop listening control, live "Xs / 30s of speech" progress, onDisappear safety stop, actionable failure copy; the standalone recorder is no longer used by this screen. - LiveGatewayBridge: enrollVoiceprintOwnerFromRecording + additive segment counts on LiveVoiceprintEnrollmentResult. - VoiceprintLiveIdentityTests: repaired 3 stale expectations to assert against the canonical injectionText(for:) instead of duplicated literals. - xcodebuild build (device, CODE_SIGNING_ALLOWED=NO): zero Swift errors. - Simulator suites after a clean DerivedData rebuild: OwnerEnrollmentModelTests (9 new) + VoiceprintLiveIdentityTests + OwnerEnrollmentTests — TEST SUCCEEDED. - bunx tsc --noEmit clean; TS voiceprint suite untouched and green. 1. Device acceptance pending: re-enroll via the new flow, then verify live recognition end-to-end. 2. The listening session uses a normal OpenAI realtime connection (silent); enrollment therefore needs network + a session's input-audio cost. 3. Guest contamination is mitigated by guidance copy only (v1). (cherry picked from commit 97a4efb)
Layer 2 of the benchmark plan (docs/voiceprint-architecture.md): simulates deterministic (seeded LCG) turn-decision sequences through the REAL A2 reducer plus the auto-scorer's short-turn neutrality gating, and compares evidence configs across four scenarios (owner-only with backchannels, owner+guest interleave, guest takeover, sparse owner with long gaps). Metrics: turns-to-establish, owner_present lost, turns-to-detect-guest, final-verdict correctness. bun run scripts/bench-voiceprint-evidence.ts prints the table. Result: the production config (owner_flip 2 / non_owner_flip 4 / min_turn_ms 2000 / stale 600s) is the only family correct on all four scenarios — both legacy symmetric configs never establish in the sparse scenario (their 60s stale default decays the streak between turns) and legacy flip=2 false-flips on owner backchannels. No neighbor variant strictly dominates it. ## Validation - bunx tsc --noEmit: clean; bun test test-bench-voiceprint-evidence.ts: 6 pass - Script runs to completion and prints the full comparison table. ## Limitations / Follow-ups 1. Decision sequences are synthetic (probabilities from measured device data); layer 1 (FAR/FRR on a real multi-speaker corpus) remains planned. (cherry picked from commit 6c74a74)
The enrollment UX needs 'continue recording': when the server counts less voiced speech than the client estimated (silences beyond the 74% heuristic), a user must be able to ADD another take rather than discard and restart — pressing the old single-recording flow again silently replaced the previous capture. recordingBaseIds (1..10, distinct, media-id-regex validated) enrolls all takes together, selected per recording in take order and fed through the same enrollment seam; the single recordingBaseId field remains accepted for back-compat. Segment counts aggregate across takes. ## Validation - bunx tsc --noEmit: clean - bun test tests/test-voiceprint-enrollment.ts: 19 pass / 0 fail (2 new: two below-floor takes enroll together after a single-take rejection; empty/oversized/duplicate id-list validation) ## Limitations / Follow-ups 1. iOS multi-take flow (contextual Continue-recording button, server-anchored progress) lands separately. (cherry picked from commit bd5a07a)
…ress User-reported UX confusion: after the server rejected a take as not_enough_speech, the screen showed a GREEN client-side "enough speech (31s)" while an orange "keep talking 14 more seconds" (server counted ~16s voiced) sat at the bottom — and the only buttons either discarded the take silently (Start listening = a fresh recording base) or explicitly (Start over). The mental model wants "Continue recording" that KEEPS what was captured. - Takes accumulate: capturedRecordingBaseIds collects one base per listening session; submit enrolls ALL takes together via recordingBaseIds (gateway bd5a07a); only Start over / reset discards. - One progress truth: a single "Xs / 30s of speech" row. After a server not_enough_speech the row anchors to the SERVER-counted voiced ms — it replaces the client estimates of every counted take (no double counting) — and shows "keep talking about N more seconds" in place, computed from server truth. The green state hedges: "About Xs captured — the server makes the final count." The contradictory bottom-of-screen message is gone. - Contextual primary button: Start listening (no takes) / Continue recording (takes exist) with "Keeps what you've recorded and adds to it." ## Validation - xcodebuild build (device, CODE_SIGNING_ALLOWED=NO): BUILD SUCCEEDED. - Simulator: OwnerEnrollmentModelTests (2 new: takes accumulate + submit in order; server anchor replaces counted takes, later takes add on) + OwnerEnrollmentTests + VoiceprintLiveIdentityTests: TEST SUCCEEDED. - bunx tsc --noEmit clean; gateway enrollment tests 19 pass (regression). ## Limitations / Follow-ups 1. Device acceptance of the full multi-take flow pending. 2. The in-progress take's contribution remains a client estimate until the server next counts it. (cherry picked from commit 0bbe29b)
(cherry picked from commit 95315dd)
Device acceptance found a template enrolled at the bare 30s floor (30.7s /
23 segments) scores live turns in the possible_owner grey band (0.74-0.76)
instead of clearing owner_accept — the owner never establishes. The manually
validated 62s live-domain template scored 0.79-0.84 on every clean turn, so
the floor is a MINIMUM, not a target. Copy now guides toward ~a minute: the
listening footer, the step-1 banner, and the green enough-state ("30s is the
minimum; closer to a minute makes recognition noticeably stronger").
## Validation
- xcodebuild build (device, CODE_SIGNING_ALLOWED=NO): BUILD SUCCEEDED.
- Copy-only change; no logic touched.
## Limitations / Follow-ups
1. Consider surfacing a soft second target in the progress row (30s gate,
60s recommended) rather than copy alone.
(cherry picked from commit 3c9ec33)
Follow-through on the device finding: a template enrolled at the bare 30s
server floor lands live turns in the possible_owner grey band and never
establishes the owner, while ~60s scores 0.79-0.84 and establishes in
seconds. Copy alone would still let users stop at the misleading 30s green
check, so the client now GATES on a 60s guided voiced target: the progress
row reads Xs / 60s, the enough state appears only past 60s ("that's enough
for strong recognition"), and keep-talking hints count down to 60s. The
server keeps accepting anything over its own 30s minimum, so a server-counted
45s take still enrolls if submitted (e.g. by an older client).
## Validation
- xcodebuild build (device): BUILD SUCCEEDED; OwnerEnrollmentModel/
OwnerEnrollment/VoiceprintLiveIdentity suites: TEST SUCCEEDED (expectations
retargeted to the 60s gate).
## Limitations / Follow-ups
1. The 60s target is client policy; consider a config-driven target if tuning
shows a different sweet spot per model.
(cherry picked from commit 521a4f8)
Cold-start latency: a fresh live session needed two consecutive hard owner turns before the identity reached the agent, so a user who opens a session with short greetings and immediately asks "do you know me?" races the evidence and gets the model's default disclaimer (observed on device at 1:15 after the 43s re-enroll: the SAME session recognized the owner at 0.79- 0.88 on every substantial turn moments later). evidence.instant_owner_confidence (production: 0.85): a single owner_speaking turn at/above this normalized confidence establishes owner_present immediately; the two-consecutive path and all overturn/stickiness semantics are unchanged, and the fast path never fires on possible_owner, scoreless observations, or when unset (default off). The auto-scorer now passes the per-turn confidence into the reducer as the observation score. Safe margin: the owner's clean turns measure 0.85+, different real speakers far below. ## Validation - bunx tsc --noEmit: clean; bun test tests/*voiceprint*: 425 pass / 0 fail (4 new: instant establish, below-bar needs streak, possible/scoreless never fast-path, unset disables) - Config parse verified end-to-end (instantOwnerConfidence reaches the tuning). ## Limitations / Follow-ups 1. The very first utterance still needs to be a substantial sentence; short greetings score in the grey band regardless. (cherry picked from commit e647cbf)
Device finding: every enrollment reported ~41s voiced no matter how long the user talked. Confirmed root cause: the 90s total segment-audio selection budget (90s x the user's measured 0.45-0.48 voiced fraction = the observed 40.8/41.4/43.1s across three consecutive enrollments, all pinned at exactly 30 segments). The cap was invisible because only the rejection path logged segment counts. - Both selection budgets (per-recording and total across takes) raised to 180s: ~85-95s voiced at the measured fraction, past which a CAM++ centroid sees diminishing returns. - The enrolled AND rejected log lines now carry the segment counts (used/capped/rejected/afterGap), so capping is visible in operations. ## Validation - bunx tsc --noEmit: clean; bun test tests/*voiceprint*: 425 pass / 0 fail (budget expectations updated). ## Limitations / Follow-ups 1. The voiced-fraction estimate (0.74 heuristic vs the user's measured ~0.47) still drives only the client-side gate; the server counts truth. (cherry picked from commit 2b4ccde)
Three device findings from the convergence pass: - The enrollment listening session leaked into the user's chat/session record. New TRANSIENT conversationJournalingEnabled config flag (never persisted; no UserDefaults key) — when false the session leaves NO trace: no app chat entries, no session-journal lines, no gateway transcript appends, no session-end memory distill. Set only by the enrollment override; every user-visible session keeps the default. The enrollment monologue is biometric capture, not conversation. - Honest success copy when the selection budget capped the takes (enrolledMessage(for:) reports what was actually used). - Simulator UX pass fixes: retry affordance when the live connection fails to come up, connecting state visibility, and layout/copy roughness found walking the flow on iPhone 17 (details in the workflow report). - xcodebuild build (device, CODE_SIGNING_ALLOWED=NO): BUILD SUCCEEDED. - Simulator: OwnerEnrollmentModelTests + OwnerEnrollmentTests + VoiceprintLiveIdentityTests + SettingsValidationTests (journaling-flag coverage): TEST SUCCEEDED. - Review loop ran 4 rounds over these changes; a fresh-eyes structural review of the whole stack then returned zero must-change items. 1. Enrollment entry point is still at the bottom of a long settings scroll (product decision deferred). 2. Simulator cannot exercise the real mic path; listening-flow states beyond connection failure verified by unit tests only. (cherry picked from commit a5b197f)
Final documentation for converged voiceprint v1, cross-checked against code: - voiceprint-architecture.md refreshed: instant-establish fast path, split gateway modules, 19 RPCs (was 17), full current config schema, multi-take silent-listening enrollment, convergence record. - voiceprint-modules.md (new): per-file structure reference across all layers (41 core modules, gateway, sidecar, iOS) — role, key exports, callers, invariants. - voiceprint-enrollment.md (new): the enrollment guide — listening-session flow, multi-take semantics, 60s target rationale with measured numbers, failure states + copy, capture-domain parity, RPC contract. ## Validation - Claims cross-checked against source by the authoring pass (RPC count and module inventory verified by grep, not memory). ## Limitations / Follow-ups 1. None. (cherry picked from commit 1f55acf)
Device timeline 2026-07-13 14:38-14:41: the enrollment listening session ran live recognition on the enrollment monologue itself — the gateway scored it and emitted owner_present DURING enrollment (14:38:43), wasted sidecar work against the very template being replaced, and primed the iOS identity machine so a subsequent same-verdict broadcast can dedupe to a no-op. The enrollment config override now sets voiceprintRealtimeEnabled=false: capture-only. Media upload is unaffected (driven independently by mediaPersistenceMode=.liveUpload, which enroll_owner_from_recording needs); only the VAD/turn event stream to the gateway stops, so nothing is scored. ## Validation - xcodebuild build (device): BUILD SUCCEEDED; OwnerEnrollmentModelTests + SettingsValidationTests: TEST SUCCEEDED. ## Limitations / Follow-ups 1. Injection latency while the assistant is speaking (floor-busy buffering, flushed on bot-stop) can still delay the identity reaching the model by a turn — observed self-healing on device; acceptable for v1. (cherry picked from commit 9bb25d7)
…ridge The reviewed owner-tag memory bridge (src/identity/voiceprint/memory-bridge.ts, from 9b25eb4) imports MemoryCandidate / MemoryCandidateAllowedUses / buildMemoryCandidate from src/memory/candidate.ts. That contract file was introduced by the memory-promotion commit 849b663, which is intentionally NOT part of this voiceprint-only branch. Rather than pull in the whole 849b663 change (distill / person-snapshot / scheduler / index.ts wiring — none of it voiceprint), add ONLY the standalone candidate.ts contract the bridge hard-depends on. The file is self-contained on main: it imports only node builtins, ../storage/config (getConfigDir) and identity/core types, all of which already exist here. This restores a clean `tsc --noEmit` without dragging non-voiceprint memory infrastructure onto the branch.
The end-to-end dataflow audit flagged a latent config coupling: production overrides staleTimeoutMs to 600s via config.json, but the code default stayed 60s — a deployment without the evidence block would silently decay a settled owner during natural pauses. The evidence-layer benchmark's sparse-owner scenario shows a 60s default NEVER establishes on sparse conversations, so the measured value becomes the default. ## Validation - bunx tsc --noEmit clean; evidence + auto-score + benchmark suites: 48 pass / 0 fail (the benchmark's legacy rows pin their own explicit values). ## Limitations / Follow-ups 1. None. (cherry picked from commit b6ddda5c59a82b28543476f529778755392eced7)
This was referenced Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
scripts/bench-voiceprint-evidence.ts) — the shipped config is the only one correct across all four simulated scenarios.docs/voiceprint-architecture.md,voiceprint-modules.md,voiceprint-enrollment.md.This branch is a clean cherry-pick of ONLY the voiceprint work from the development branch (55 commits, authorship preserved,
-xannotated): no unrelated features, no local signing anywhere in history, one minimal adapter commit vendoring theMemoryCandidatecontract the A9 memory bridge consumes.Validation
bunx tsc --noEmitclean;bun test tests/*voiceprint*: 422 pass / 0 fail (2 env-gated ONNX e2e skips without the model fixture; they pass with it).src/identity/voiceprint,src/gateway/voiceprint-*,services/voiceprint,docs/voiceprint-*).Limitations / Follow-ups