Skip to content

Commit ee55376

Browse files
SportingerSportinger
committed
feat: execute transcript assemblies through kernel programs
1 parent 1f1d99f commit ee55376

11 files changed

Lines changed: 219 additions & 18 deletions

File tree

‎docs/ongoing/FlashBoard-Agent-Module-Foundation.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -123,10 +123,11 @@ Die Tabelle hier ist die menschenlesbare Spiegelung:
123123

124124
| Modul | Bundle | Operationen | Status | Akzeptanz-Aufgabe |
125125
|---|---|---|---|---|
126-
| remove-ranges | `fast-v2-timeline-editing-2026-08-02` | `timeline.segment.split.v1`, `timeline.segment.delete-many.v1`, `timeline.visual.capture-grid.v1` | live (Referenzmodul, grandfathered ohne Preview/Commit-Paar) | Bereiche aus einem Clip entfernen, optional visuell prüfen |
127-
| timeline-intercut | `fast-v2-timeline-editing-2026-08-02` | `timeline.intercut.preview.v1`, `timeline.segment.split.v1`, `timeline.segment.delete-many.v1`, `timeline.intercut.commit.v1` | live | Transkript-getaktete Shots aus mehreren Takes atomar abwechseln und verknüpften Ton erhalten |
128-
| editable-hook | `fast-v2-timeline-editing-2026-08-02` | `timeline.hook.preview.v1`, `timeline.hook.commit.v1`, `timeline.hook.refine.commit.v1`, `timeline.visual.capture-grid.v1` | live | Editierbaren Text-Hook mit nativen Backplates schnell erzeugen und danach automatisch einen Kontrollframe liefern |
129-
| editable-hook-refinement | `fast-v2-timeline-editing-2026-08-02` | `timeline.hook.preview.v1`, `timeline.hook.commit.v1`, `timeline.hook.refine.commit.v1`, `timeline.visual.capture-grid.v1` | live | Einzelne Text-/Backplate-Zeilen über stabilen `hookId` und `rowIndex` nachbessern, nach jeder Änderung neu rendern, maximal zwei Refinements |
126+
| remove-ranges | `fast-v2-transcript-assembly-2026-08-03` | `timeline.segment.split.v1`, `timeline.segment.delete-many.v1`, `timeline.visual.capture-grid.v1` | live (Referenzmodul, grandfathered ohne Preview/Commit-Paar) | Bereiche aus einem Clip entfernen, optional visuell prüfen |
127+
| timeline-intercut | `fast-v2-transcript-assembly-2026-08-03` | `timeline.intercut.preview.v1`, `timeline.segment.split.v1`, `timeline.segment.delete-many.v1`, `timeline.intercut.commit.v1` | live | Transkript-getaktete Shots aus mehreren Takes atomar abwechseln und verknüpften Ton erhalten |
128+
| transcript-assembly | `fast-v2-transcript-assembly-2026-08-03` | `timeline.editor.program.preview.v1`, `timeline.editor.program.commit.v1` | live | Verbatim Text in Wort-, Phrasen- oder Satzgranularität deklarieren; der private Kernel löst Zeitstempel auf und kompiliert eine neue Komposition aus atomaren Segment-Inserts. Die öffentliche Grenze bleibt ein generisches Edit-Programm ohne Transkript-Orchestrierung. |
129+
| editable-hook | `fast-v2-transcript-assembly-2026-08-03` | `timeline.hook.preview.v1`, `timeline.hook.commit.v1`, `timeline.hook.refine.commit.v1`, `timeline.visual.capture-grid.v1` | live | Editierbaren Text-Hook mit nativen Backplates schnell erzeugen und danach automatisch einen Kontrollframe liefern |
130+
| editable-hook-refinement | `fast-v2-transcript-assembly-2026-08-03` | `timeline.hook.preview.v1`, `timeline.hook.commit.v1`, `timeline.hook.refine.commit.v1`, `timeline.visual.capture-grid.v1` | live | Einzelne Text-/Backplate-Zeilen über stabilen `hookId` und `rowIndex` nachbessern, nach jeder Änderung neu rendern, maximal zwei Refinements |
130131

131132
## 8. Einstiegspunkte und Regeln
132133

‎docs/ongoing/FlashBoard-Kernel-Agent-ChatCut-Plan.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -654,8 +654,8 @@ nicht wiederholt nach Informationen, die nur wegen eines laufenden Jobs fehlen.
654654
Der öffentliche Fast-V2-Vertrag erhält sinngemäß:
655655

656656
```ts
657-
'timeline.transcript-assembly.preview.v1'
658-
'timeline.transcript-assembly.commit.v1'
657+
'timeline.editor.program.preview.v1'
658+
'timeline.editor.program.commit.v1'
659659
```
660660

661661
Der lokale Executor kann darunter bestehende Split-, Move-, Delete-, Ripple-

‎functions/lib/hostedAgent/route.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -409,6 +409,8 @@ const WP1_OPERATION_IDS = new Set([
409409
'timeline.intercut.preview.v1',
410410
'timeline.segment.delete-many.v1',
411411
'timeline.segment.split.v1',
412+
'timeline.editor.program.commit.v1',
413+
'timeline.editor.program.preview.v1',
412414
'timeline.visual.capture-grid.v1',
413415
]);
414416

@@ -426,6 +428,7 @@ function validProjectedOperationResult(value: unknown, operationId: string): boo
426428
operationId === 'timeline.editor.destructive.v1'
427429
|| operationId === 'timeline.editor.inspect.v1'
428430
|| operationId === 'timeline.editor.mutate.v1'
431+
|| operationId === 'timeline.editor.program.commit.v1'
429432
) {
430433
if (value.data === undefined) return true;
431434
try {
@@ -445,6 +448,7 @@ function validProjectedOperationResult(value: unknown, operationId: string): boo
445448
|| operationId === 'timeline.intercut.commit.v1'
446449
|| operationId === 'timeline.intercut.preview.v1'
447450
|| operationId === 'timeline.segment.delete-many.v1'
451+
|| operationId === 'timeline.editor.program.preview.v1'
448452
) return value.data === undefined;
449453
if (!isRecord(value.data)) return false;
450454
if (operationId === 'timeline.segment.split.v1') {

‎src/services/aiTools/definitions/clips.ts‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -321,8 +321,8 @@ export const clipToolDefinitions: ToolDefinition[] = [
321321
description: 'ID of the media file in the media pool (from getMediaItems)',
322322
},
323323
trackId: {
324-
type: 'string',
325-
description: 'ID of the track to add the clip to',
324+
type: ['string', 'null'],
325+
description: 'ID of the track to add the clip to, or null to select the first compatible active-composition track',
326326
},
327327
startTime: {
328328
type: 'number',
@@ -336,6 +336,12 @@ export const clipToolDefinitions: ToolDefinition[] = [
336336
type: 'number',
337337
description: 'End time within the source file in seconds',
338338
},
339+
deClickFadeSeconds: {
340+
type: 'number',
341+
minimum: 0,
342+
maximum: 0.02,
343+
description: 'Optional automatic audio fade duration on both inserted segment edges (0-0.02 seconds)',
344+
},
339345
},
340346
required: ['mediaFileId', 'trackId', 'startTime', 'inPoint', 'outPoint'],
341347
},

‎src/services/aiTools/handlers/clips/addSegment.ts‎

Lines changed: 73 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,13 @@ import {
88
import {
99
MIN_CLIP_DURATION,
1010
} from '../../../../stores/timeline/editOperations/trimOperations';
11+
import type { ClipAudioEditOperation } from '../../../../types';
12+
import { clearProcessedAudioAnalysisRefs } from '../../../../stores/timeline/helpers/audioAnalysisStateHelpers';
13+
import { createAudioEditOperationId } from '../../../../stores/timeline/audioEdit/audioEditHelpers';
14+
import {
15+
createAutomaticCutDeClickOperation,
16+
MAX_AUTOMATIC_DE_CLICK_FADE_SECONDS,
17+
} from '../../../audio/automaticCutDeClick';
1118

1219
export function insertedClipAlreadyMatchesRequestedSegment(
1320
clip: { inPoint: number; outPoint: number },
@@ -18,6 +25,18 @@ export function insertedClipAlreadyMatchesRequestedSegment(
1825
&& Math.abs(clip.outPoint - outPoint) < 1e-6;
1926
}
2027

28+
export function resolveAddClipSegmentTrackId(
29+
requestedTrackId: string | null,
30+
mediaType: string,
31+
tracks: readonly { id: string; type: string }[],
32+
): string | undefined {
33+
if (requestedTrackId !== null) {
34+
return tracks.find((track) => track.id === requestedTrackId)?.id;
35+
}
36+
const preferredTrackType = mediaType === 'audio' ? 'audio' : 'video';
37+
return tracks.find((track) => track.type === preferredTrackType)?.id;
38+
}
39+
2140
/**
2241
* Add a clip segment from the media pool with specific in/out points.
2342
* Self-contained handler — fetches both stores internally.
@@ -26,10 +45,14 @@ export async function handleAddClipSegment(
2645
args: Record<string, unknown>,
2746
): Promise<ToolResult> {
2847
const mediaFileId = args.mediaFileId as string;
29-
const trackId = args.trackId as string;
48+
const requestedTrackId = typeof args.trackId === 'string' ? args.trackId : null;
3049
const startTime = args.startTime as number;
3150
const inPoint = args.inPoint as number;
3251
const outPoint = args.outPoint as number;
52+
const deClickFadeSeconds = typeof args.deClickFadeSeconds === 'number'
53+
&& Number.isFinite(args.deClickFadeSeconds)
54+
? Math.max(0, Math.min(MAX_AUTOMATIC_DE_CLICK_FADE_SECONDS, args.deClickFadeSeconds))
55+
: 0;
3356

3457
if (inPoint >= outPoint) {
3558
return { success: false, error: 'inPoint must be less than outPoint' };
@@ -58,10 +81,21 @@ export async function handleAddClipSegment(
5881
return { success: false, error: `File object not available for media: ${mediaFileId}. Try re-importing the file.` };
5982
}
6083

61-
// Validate track
62-
const track = timelineStore.tracks.find(t => t.id === trackId);
63-
if (!track) {
64-
return { success: false, error: `Track not found: ${trackId}` };
84+
// A null track id is the deterministic runtime binding used by private
85+
// kernel edit programs after creating and opening a destination composition.
86+
const preferredTrackType = mediaFile.type === 'audio' ? 'audio' : 'video';
87+
const trackId = resolveAddClipSegmentTrackId(
88+
requestedTrackId,
89+
mediaFile.type,
90+
timelineStore.tracks,
91+
);
92+
if (!trackId) {
93+
return {
94+
success: false,
95+
error: requestedTrackId === null
96+
? `No compatible ${preferredTrackType} track is available in the active composition`
97+
: `Track not found: ${requestedTrackId}`,
98+
};
6599
}
66100

67101
const mutationSnapshot = captureMutationEntitySnapshot('clip', timelineStore.clips);
@@ -109,12 +143,46 @@ export async function handleAddClipSegment(
109143
if (clip.linkedClipId) trimmedClipIds.add(clip.linkedClipId);
110144
}
111145

146+
let deClickFadesApplied = 0;
147+
if (deClickFadeSeconds > 0) {
148+
const newClipIds = new Set(newClips.map((clip) => clip.id));
149+
const audioTrackIds = new Set(
150+
useTimelineStore.getState().tracks
151+
.filter((candidate) => candidate.type === 'audio')
152+
.map((candidate) => candidate.id),
153+
);
154+
useTimelineStore.setState((state) => ({
155+
clips: state.clips.map((clip) => {
156+
if (!newClipIds.has(clip.id) || !audioTrackIds.has(clip.trackId)) return clip;
157+
const operations = (['in', 'out'] as const)
158+
.map((edge) => createAutomaticCutDeClickOperation(
159+
clip,
160+
edge,
161+
deClickFadeSeconds,
162+
{ createdAt: Date.now(), id: createAudioEditOperationId() },
163+
))
164+
.filter((operation): operation is ClipAudioEditOperation => operation !== null);
165+
if (operations.length === 0) return clip;
166+
deClickFadesApplied += operations.length;
167+
return clearProcessedAudioAnalysisRefs({
168+
...clip,
169+
audioState: {
170+
...(clip.audioState ?? {}),
171+
editStack: [...(clip.audioState?.editStack ?? []), ...operations],
172+
},
173+
});
174+
}),
175+
}));
176+
if (deClickFadesApplied > 0) useTimelineStore.getState().invalidateCache();
177+
}
178+
112179
// Return info about created clips
113180
const createdClips = useTimelineStore.getState().clips.filter(c => newClips.some(n => n.id === c.id));
114181
return {
115182
success: true,
116183
data: {
117184
clipCount: createdClips.length,
185+
deClickFadesApplied,
118186
clips: createdClips.map(c => ({
119187
id: c.id,
120188
trackId: c.trackId,

‎src/services/kernelClient/hostedAgent/fastV2FetchTransport.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -742,6 +742,7 @@ function validProjectedOperationResult(value: unknown, operationId: string): boo
742742
operationId === 'timeline.editor.destructive.v1'
743743
|| operationId === 'timeline.editor.inspect.v1'
744744
|| operationId === 'timeline.editor.mutate.v1'
745+
|| operationId === 'timeline.editor.program.commit.v1'
745746
) {
746747
if (value.data === undefined) return true;
747748
try {
@@ -761,6 +762,7 @@ function validProjectedOperationResult(value: unknown, operationId: string): boo
761762
|| operationId === 'timeline.intercut.commit.v1'
762763
|| operationId === 'timeline.intercut.preview.v1'
763764
|| operationId === 'timeline.segment.delete-many.v1'
765+
|| operationId === 'timeline.editor.program.preview.v1'
764766
) return value.data === undefined;
765767
if (!isRecord(value.data)) return false;
766768
if (operationId === 'timeline.segment.split.v1') {

‎src/services/kernelClient/hostedAgent/fastV2StartContract.ts‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,16 +8,16 @@ export const HOSTED_AGENT_FAST_V2_PROTOCOL_VERSION = 'fast-agent-v2' as const;
88
// former eight-call batching default. Keep a finite fail-closed ceiling.
99
export const HOSTED_AGENT_FAST_V2_MAX_TOOL_CALLS_PER_ROUND = 256 as const;
1010
export const HOSTED_AGENT_FAST_V2_EDITOR_TOOL_CATALOG_DIGEST =
11-
'sha256:39eabfa69950ece55280a7fc77e091d98e11b62aa8da869db57ae6f8fb3c2d05' as const;
11+
'sha256:0418657c23c5bbef2e5a9c270c4793208f5045201aee6b5a9bb2ec22855636ff' as const;
1212
export const HOSTED_AGENT_FAST_V2_EXECUTION_CONTRACT_VERSION =
1313
PUBLIC_OPERATION_CONTRACT_V1.contractVersion;
1414
export const HOSTED_AGENT_FAST_V2_EXECUTION_CONTRACT_DIGEST =
1515
PUBLIC_OPERATION_CONTRACT_DIGEST_V1;
1616
export const HOSTED_AGENT_FAST_V2_SERVICE_ENVELOPE_VERSION = 1 as const;
1717
export const HOSTED_AGENT_FAST_V2_PROMPT_VERSION =
18-
'fast-v2-prompt-tool-recovery-2026-08-03' as const;
18+
'fast-v2-prompt-transcript-assembly-2026-08-03' as const;
1919
export const HOSTED_AGENT_FAST_V2_CAPABILITY_BUNDLE_VERSION =
20-
'fast-v2-progressive-editor-tools-captions-2026-08-02' as const;
20+
'fast-v2-transcript-assembly-2026-08-03' as const;
2121
export const HOSTED_AGENT_FAST_V2_MODEL_POLICY_VERSION =
2222
'fast-v2-model-policy-2026-08-02' as const;
2323
export const HOSTED_AGENT_FAST_V2_BUDGET_POLICY_VERSION =

‎src/services/kernelClient/wp1Spike/editorOperationDispatcher.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ function genericOperationRisk(
2828
if (operationId === 'timeline.editor.inspect.v1') return 'read-only';
2929
if (operationId === 'timeline.editor.mutate.v1') return 'mutating';
3030
if (operationId === 'timeline.editor.destructive.v1') return 'destructive';
31+
if (operationId === 'timeline.editor.program.commit.v1') return 'mutating';
3132
return undefined;
3233
}
3334

‎src/services/kernelClient/wp1Spike/publicOperationContracts.ts‎

Lines changed: 57 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@ export type PublicOperationIdV1 =
1010
| 'timeline.intercut.preview.v1'
1111
| 'timeline.segment.delete-many.v1'
1212
| 'timeline.segment.split.v1'
13+
| 'timeline.editor.program.commit.v1'
14+
| 'timeline.editor.program.preview.v1'
1315
| 'timeline.visual.capture-grid.v1';
1416

1517
export type PublicOperationEffectV1 =
@@ -266,6 +268,59 @@ export const PUBLIC_OPERATION_CONTRACT_V1 = {
266268
risk: 'mutating',
267269
transaction: 'none',
268270
},
271+
{
272+
arguments: {
273+
additionalProperties: false,
274+
fields: {
275+
requestJson: {
276+
kind: 'bounded-json-string-v1',
277+
maximumLength: 100_000,
278+
minimumLength: 2,
279+
required: true,
280+
},
281+
},
282+
schemaVersion: 1,
283+
},
284+
confirmation: 'inherit',
285+
dispatcher: {
286+
adapterVersion: 'ms-kernel-ai-tool-dispatcher-v1',
287+
callerContext: 'kernel',
288+
suppressHistory: false,
289+
toolName: 'executeKernelEditorTool',
290+
},
291+
effects: ['mediaDuration', 'segmentation', 'sourceCoverage', 'sourceOrder', 'timelinePlacement'],
292+
id: 'timeline.editor.program.commit.v1',
293+
result: {
294+
bindable: [],
295+
projection: { kind: 'bounded-json-v1', maximumCharacters: 500_000 },
296+
schemaVersion: 1,
297+
},
298+
risk: 'mutating',
299+
transaction: 'none',
300+
},
301+
{
302+
arguments: {
303+
additionalProperties: false,
304+
fields: {},
305+
schemaVersion: 1,
306+
},
307+
confirmation: 'inherit',
308+
dispatcher: {
309+
adapterVersion: 'ms-kernel-ai-tool-dispatcher-v1',
310+
callerContext: 'kernel',
311+
suppressHistory: true,
312+
toolName: 'getTimelineState',
313+
},
314+
effects: [],
315+
id: 'timeline.editor.program.preview.v1',
316+
result: {
317+
bindable: [],
318+
projection: { kind: 'success-only-v1' },
319+
schemaVersion: 1,
320+
},
321+
risk: 'read-only',
322+
transaction: 'none',
323+
},
269324
{
270325
arguments: {
271326
additionalProperties: false,
@@ -582,7 +637,7 @@ export const PUBLIC_OPERATION_CONTRACT_V1 = {
582637

583638
// Recomputed by the WP1 compatibility check from the canonical JSON above.
584639
export const PUBLIC_OPERATION_CONTRACT_DIGEST_V1 =
585-
'sha256:04e98a757c065eac124ffcd6e035a714d3714dfa2c87152eaf9307cf4133b007' as const;
640+
'sha256:75c941fa67c87e6f14f2933cec13499cbce5b0d5e164c004ec10b002d4c8baaa' as const;
586641

587642
export const PUBLIC_COMPILED_PLAN_EXTENSION_V1 = {
588643
bindingVersion: 'ms-prior-result-path-v1',
@@ -596,7 +651,7 @@ export const PUBLIC_COMPILED_PLAN_EXTENSION_V1 = {
596651
} as const;
597652

598653
export const PUBLIC_COMPILED_PLAN_DIGEST_V1 =
599-
'sha256:da80771d51db4492c7a133b48f63c04d7df3a0061a50e39ad8f9c42be42284ca' as const;
654+
'sha256:fddab5db3c573719c9e8200bd343a89eca3175d96087d422e9a0d7bb1b6bdfa7' as const;
600655

601656
const OPERATION_BY_ID = new Map<PublicOperationIdV1, PublicOperationSpecV1>(
602657
PUBLIC_OPERATION_CONTRACT_V1.operations.map((operation) => [operation.id, operation]),

‎tests/unit/aiToolAddClipSegment.test.ts‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ import { describe, expect, it } from 'vitest';
22
import {
33
handleAddClipSegment,
44
insertedClipAlreadyMatchesRequestedSegment,
5+
resolveAddClipSegmentTrackId,
56
} from '../../src/services/aiTools/handlers/clips/addSegment';
67

78
describe('addClipSegment preflight', () => {
@@ -30,4 +31,16 @@ describe('addClipSegment preflight', () => {
3031
10,
3132
)).toBe(false);
3233
});
34+
35+
it('binds a null track id to the first compatible active-composition track', () => {
36+
const tracks = [
37+
{ id: 'audio-1', type: 'audio' },
38+
{ id: 'video-1', type: 'video' },
39+
{ id: 'video-2', type: 'video' },
40+
];
41+
expect(resolveAddClipSegmentTrackId(null, 'video', tracks)).toBe('video-1');
42+
expect(resolveAddClipSegmentTrackId(null, 'audio', tracks)).toBe('audio-1');
43+
expect(resolveAddClipSegmentTrackId('video-2', 'video', tracks)).toBe('video-2');
44+
expect(resolveAddClipSegmentTrackId('missing', 'video', tracks)).toBeUndefined();
45+
});
3346
});

0 commit comments

Comments
 (0)