-
Notifications
You must be signed in to change notification settings - Fork 17
Expand file tree
/
Copy pathindex.html
More file actions
2908 lines (2259 loc) · 189 KB
/
Copy pathindex.html
File metadata and controls
2908 lines (2259 loc) · 189 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Voice AI & Voice Agents | An Illustrated Primer</title>
<meta name="description" content="A comprehensive guide to voice AI in 2026">
<meta property="og:title" content="Voice AI & Voice Agents | An Illustrated Primer">
<meta property="og:description" content="A comprehensive guide to voice AI in 2026">
<meta property="og:url" content="https://voiceaiandvoiceagents.com/">
<meta property="og:image" content="https://voiceaiandvoiceagents.com/images/meta.jpg">
<meta property="og:image:secure_url" content="https://voiceaiandvoiceagents.com/images/meta.jpg">
<meta property="og:image:type" content="image/jpeg">
<meta property="og:image:width" content="1980">
<meta property="og:image:height" content="1114">
<meta property="og:image:alt" content="Voice AI & Voice Agents — An Illustrated Primer">
<meta property="og:type" content="website">
<meta name="twitter:card" content="summary_large_image">
<meta name="twitter:title" content="Voice AI & Voice Agents | An Illustrated Primer">
<meta name="twitter:description" content="A comprehensive guide to voice AI in 2026">
<meta name="twitter:image" content="https://voiceaiandvoiceagents.com/images/meta.jpg">
<link rel="stylesheet" href="styles.css">
<link rel="icon" href="images/favicon.ico">
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=DM+Sans:ital,wght@0,400;0,500;0,700;1,400&display=swap" rel="stylesheet">
<link href="https://fonts.googleapis.com/css2?family=DM+Mono:wght@400;500&display=swap" rel="stylesheet">
<link rel="stylesheet" href="https://use.typekit.net/rff5lbb.css">
<link rel="stylesheet" href="script/binary-numbers.css">
<style>
#table-of-contents ul {
list-style-type: none;
padding-left: 20px;
}
</style>
</head>
<body>
<div style="position: absolute; top: 1rem; right: 2rem;">
<a href="/ja" style="color: var(--foreground); text-decoration: none; font-weight: 500;">Japanese</a>
</div>
<div class="container">
<div style="float: right; margin: 1rem 0 1rem 1rem; width: 10rem;">
<img src="images/Figure 0100 horizontal.svg">
</div>
<header class="header">
<h1 class="title">Voice AI & Voice Agents</h1>
<h2 class="subtitle">An Illustrated Primer</h2>
</header>
<nav id="table-of-contents">
<h2 class="table-of-contents-title">Table of Contents</h2>
<ol>
<li><a href="#conversational-voice-ai">Conversational Voice AI in 2026</a></li>
<li><a href="#voice-ai-not-just-voice">Voice AI is not just "voice"</a></li>
<li>
<a href="#about-this-guide">About this guide</a>
<ul>
<li><a href="#benchmarks-notes">3.1. Keeping up with model releases</a></li>
</ul>
</li>
<li><a href="#basic-loop">The basic conversational AI loop</a></li>
<li>
<a href="#core-tech">Core technologies and best practices</a>
<ul>
<li><a href="#latency">5.1. Latency</a></li>
<li>
<a href="#llms-for-voice">5.2. LLMs for voice use cases</a>
<ul>
<li><a href="#cost-comparison">5.2.1. Cost comparison</a></li>
<li><a href="#open-source">5.2.2. Open source / open weights</a></li>
<li><a href="#speech-to-speech">5.2.3. What about speech-to-speech models?</a></li>
</ul>
</li>
<li>
<a href="#benchmarks">5.3. LLM benchmarks</a>
<ul>
<li><a href="#pareto-frontier-llm">5.3.1. The intelligence/latency Pareto frontier</a></li>
</ul>
</li>
<li>
<a href="#speech-to-text">5.4. Speech-to-text</a>
<ul>
<li><a href="#pareto-frontier">5.4.1. The STT Pareto frontier</a></li>
<li><a href="#prompting-help">5.4.2. Prompting can help the LLM</a></li>
</ul>
</li>
<li>
<a href="#text-to-speech">5.5. Text-to-speech</a>
</li>
<li>
<a href="#audio-processing">5.6. Audio processing</a>
<ul>
<li><a href="#microphones-agc">5.6.1. Microphones and automatic gain control</a></li>
<li><a href="#echo-cancellation">5.6.2. Echo cancellation</a></li>
<li><a href="#noise-suppression">5.6.3. Noise suppression, speech, and music</a></li>
<li><a href="#encoding">5.6.4. Encoding</a></li>
<li><a href="#server-side-noise">5.6.5. Server-side noise processing and speaker isolation</a></li>
<li><a href="#voice-activity-detection">5.6.6. Voice activity detection</a></li>
</ul>
</li>
<li>
<a href="#network-transport">5.7. Network transport</a>
<ul>
<li><a href="#websockets-webrtc">5.7.1. WebSockets and WebRTC</a></li>
<li><a href="#http">5.7.2. HTTP</a></li>
<li><a href="#quic-moq">5.7.3. QUIC and MoQ</a></li>
<li><a href="#network-routing">5.7.4. Network routing</a></li>
</ul>
</li>
<li>
<a href="#turn-detection">5.8. Turn detection</a>
<ul>
<li><a href="#voice-activity-detection-4-7">5.8.1. Voice activity detection</a></li>
<li><a href="#push-to-talk">5.8.2. Push-to-talk</a></li>
<li><a href="#endpoint-markers">5.8.3. Endpoint markers</a></li>
<li><a href="#context-aware-turn-detection">5.8.4. Context-aware turn detection (semantic VAD and smart turn)</a></li>
<li><a href="#sota-turn-detection">5.8.5. State of the art turn detection: VAD, Smart Turn, and LLM single-token tagging</a></li>
</ul>
</li>
<li>
<a href="#interruption-handling">5.9. Interruption handling</a>
<ul>
<li><a href="#avoiding-spurious-interruptions">5.9.1. Avoiding spurious interruptions</a></li>
<li><a href="#maintaining-accurate-context">5.9.2. Maintaining accurate context after an interruption</a></li>
</ul>
</li>
<li>
<a href="#managing-conversation-context">5.10. Managing conversation context</a>
<ul>
<li><a href="#differences-between-llm-apis">5.10.1. Differences between LLM APIs</a></li>
<li><a href="#modifying-context-between-turns">5.10.2. Modifying the context between turns</a></li>
</ul>
</li>
<li>
<a href="#function-calling">5.11. Function calling</a>
<ul>
<li><a href="#function-calling-reliability">5.11.1. Function calling reliability in the voice AI context</a></li>
<li><a href="#latency-function-calls">5.11.2. Function call latency</a></li>
<li><a href="#handling-interruptions">5.11.3. Handling interruptions</a></li>
<li><a href="#streaming-mode">5.11.4. Streaming mode and function call chunks</a></li>
<li><a href="#execute-function-calls">5.11.5. How and where to execute function calls</a></li>
<li><a href="#async-function-calls">5.11.6. Asynchronous function calls</a></li>
<li><a href="#parallel-composite-function-calling">5.11.7. Parallel and composite function calling</a></li>
</ul>
</li>
<li><a href="#multimodality">5.12. Multimodality</a></li>
</ul>
</li>
<li>
<a href="#multiple-models">Using multiple AI models</a>
<ul>
<li><a href="#fine-tuned-models">6.1. Using several fine-tuned models</a></li>
<li><a href="#async-inference-tasks">6.2. Performing async inference tasks</a></li>
<li><a href="#content-guardrails">6.3. Content guardrails</a></li>
<li><a href="#single-inference-actions">6.4. Performing single inference actions</a></li>
<li><a href="#self-improving-systems">6.5. Towards self-improving systems</a></li>
</ul>
</li>
<li>
<a href="#scripting">Scripting and instruction following</a>
</li>
<li>
<a href="#evals">Voice AI Evals</a>
<ul>
<li><a href="#evals-different">8.1. Voice AI evals are different from software unit tests</a></li>
<li><a href="#failure-modes">8.2. Failure modes</a></li>
<li><a href="#eval-strategy">8.3. Crafting an eval strategy</a></li>
</ul>
</li>
<li><a href="#telephony">Integrating with telephony infrastructure</a></li>
<li>
<a href="#rag-memory">RAG and memory</a>
</li>
<li>
<a href="#hosting">Hosting and scaling</a>
<ul>
<li><a href="#hosting-architecture">11.1. Architecture</a></li>
<li><a href="#hosting-cost">11.2. Calculating per-minute cost</a></li>
</ul>
</li>
<li>
<a href="#future">What's coming in 2026 and 2027</a>
</li>
<li><a href="#contributors">Contributors</a></li>
</ol>
</nav>
<main>
<div class="chunk-row">
<div class="chunk-content">
<h1 id="conversational-voice-ai">1. Conversational Voice AI in 2026</h1>
<p>LLMs are good conversationalists.</p>
<p>If you've spent much time in free-form dialog with ChatGPT or Claude, you have an intuitive sense that talking to an LLM feels quite natural and is broadly useful.</p>
<p>LLMs are also good at turning unstructured information into structured data.<sup>[1]</sup></p>
<p>Voice AI agents leverage these two LLM capabilities – conversation, and extracting structure from unstructured data – to create a new kind of user experience.</p>
</div>
<div class="chunk-notes">
<div class="chunk-footnotes">
<div class="footnote">
<p>[1] Here we mean this broadly, rather than in the narrow sense of the "structured output" feature of some LLMs.</p>
</div>
</div>
</div>
</div>
<div class="chunk-row">
<div class="chunk-content">
<p>Voice AI is being deployed today in a wide range of business contexts. For example:</p>
<ul class="arrow-list">
<li>collecting patient data prior to healthcare appointments,</li>
<li>following up on inbound sales leads,</li>
<li>handling an increasing variety of call center tasks,</li>
<li>job interviews and user research interviews,</li>
<li>coordinating scheduling and logistics between companies, and</li>
<li>answering the phone for nearly every kind of small business.</li>
</ul>
<p>On the consumer side, conversational voice (and video) AI is also starting to make its way into social applications and games. And developers are sharing personal voice AI projects and experiments every day on GitHub and social media.</p>
</div>
</div>
<div class="chunk-row">
<div class="chunk-content">
<h1 id="voice-ai-not-just-voice">2. Voice AI is not just "voice"</h1>
<p>Those of us who work on Voice AI have been thinking about:</p>
<ul class="arrow-list">
<li>multi-modal agents,</li>
<li>multi-model orchestration,</li>
<li>asynchronous tool calling,</li>
<li>context compaction,</li>
<li>hybrid local/cloud inference,</li>
<li>progressive "skills" loading,</li>
<li>agent memory,</li>
<li>continual learning,</li>
<li>sandboxes, and</li>
<li>dynamically generated user interfaces</li>
</ul>
<p>since … 2023!</p>
<p>It turns out that to build a good voice agent we needed to figure out a lot of the things that are core to building good agents in general.</p>
<p>And we had to do it while optimizing for ultra low-latency responsiveness. (Humans want voice agents to respond to them as fast as other humans do, in conversation.)</p>
<p>Today, we're seeing voice AI expand beyond the enterprise use cases that were the first large-scale adopters of agents.</p>
<p>We're building new abstractions for "thinking fast and slow" styles of agent orchestration. Teams building real-time video models are making the same kind of progress we saw voice models making in 2023 and 2024. More and more people are using voice input every day to interact with their desktops, coding agents, and co-pilots of all kinds.</p>
<p>In this guide, we'll mostly focus on the core enabling technology of "voice agents." But if you're interested in new agent architectures, new user interfaces, and multi-agent orchestrations, the global voice AI developer community is doing lots of innovative work.</p>
</div>
<div class="chunk-notes">
<div class="chapter-image">
<a href="images/voice-agent-xml-pattern.svg"><img src="images/voice-agent-xml-pattern.svg" class="image-hide-narrow" width="100%"></a>
<p class="image-caption image-hide-narrow">Voice Agent Pattern: Long-running tasks producing interleaved XML events</p>
</div>
<div class="chapter-image">
<a href="images/pipecat-distributed-agents.png"><img src="images/pipecat-distributed-agents.png" class="image-hide-narrow" width="100%" alt="Pipecat documentation: Distributed Agents — running agent processes across machines connected to the same bus"></a>
<p class="image-caption image-hide-narrow">Today's voice agents are often architecturally complex, using approaches like subagents, hybrid local/cloud inference, and dynamic context engineering.</p>
</div>
</div>
</div>
<div class="chunk-row">
<div class="chunk-content">
<h1 id="about-this-guide">3. About this guide</h1>
<p>This guide is a snapshot of the voice AI state of the art.<sup>[2]</sup></p>
<p>Building production-ready voice agents is complicated. Many elements are non-trivial to implement from scratch. If you build voice AI apps, you'll likely rely on a framework for many of the things discussed in this document. But we think it's useful to understand how the pieces fit together, whether you are building them all from scratch or not.</p>
<p>This guide was inspired by Sean DuBois' open-source book <a href="https://webrtcforthecurious.com" target="_blank">WebRTC For the Curious</a>. That book has helped numerous developers get up to speed with WebRTC since it was first released four years ago.<sup>[3]</sup></p>
<p>The voice AI code examples in this document use the <a href="https://pipecat.ai" target="_blank">Pipecat</a> open source framework.<sup>[4]</sup> Pipecat is the most widely used voice AI framework, with teams at AWS and NVIDIA, all of the large AI labs, Fortune 500 companies like ServiceNow, and thousands of startups, scale-ups, and individual developers leveraging and contributing to the codebase.</p>
<p>We've tried to give general advice in this document, rather than recommend commercial products and services. Where we highlight specific vendors, we do so because they are used by a large percentage of voice AI developers.</p>
<h2 id="benchmarks-notes">3.1. Keeping up with model releases</h2>
<p>Keeping up with the fast-changing AI model landscape is a big challenge. There is a constant stream of model releases that are relevant to voice agents.</p>
<p>Benchmarks are an important way to track model progress and figure out whether to spend time fully evaluating a specific model. Unfortunately, most of the widely publicized model benchmarks aren't useful for voice AI. Voice agents are always multi-turn, so benchmarks that do not test long, multi-turn performance won't tell you how a model performs in real conversations. Latency is very important for voice agents, and most benchmarks don't report TTFT (time to first token) or similar metrics.</p>
<p>Here are the benchmarks we use to track model releases and first-party API performance:</p>
<ul class="arrow-list">
<li><a href="https://github.com/pipecat-ai/stt-benchmark">Speech-to-text model performance benchmark</a></li>
<li><a href="https://github.com/kwindla/aiewf-eval">AIEWF LLM voice agent benchmark</a> (30-turn conversation with tool calling)</li>
<li><a href="https://benchmarks.coval.ai/overview">Coval text-to-speech and speech-to-text benchmarks</a></li>
<li><a href="https://servicenow.github.io/eva/#results">ServiceNow EVA model evaluation framework</a></li>
<li><a href="https://humannessindex.vapi.ai/">VAPI voice model humanness index</a></li>
</ul>
</div>
<div class="chunk-notes">
<div class="chunk-footnotes">
<div class="footnote">
<p>[2] We originally wrote this guide for the AI Engineering Summit in February 2025. This update was published in June 2026.</p>
</div>
<div class="footnote">
<p>[3] <a href="https://webrtcforthecurious.com" target="_blank">webrtcforthecurious.com</a> — WebRTC is relevant to voice AI, as we'll discuss later in the <a href="#websockets-webrtc">WebSockets and WebRTC</a> section.</p>
</div>
<div class="footnote">
<p>[4] Pipecat has integrations for more than 100 <a href="https://docs.pipecat.ai/server/services/supported-services" target="_blank">AI models and services</a>, along with state of-the-art implementations of things like turn detection and interruption handling. You can write code with Pipecat that uses WebSockets, WebRTC, HTTP, and telephony to communicate with users. Pipecat includes transport implementations for a variety of infrastructure platforms including Twilio, Telnyx, LiveKit, Daily, and others. There are <a href="https://docs.pipecat.ai/client/introduction" target="_blank">client-side Pipecat SDKs</a> for JavaScript, React, iOS, Android, and C++.</p>
</div>
</div>
</div>
</div>
<div class="chunk-row">
<div class="chunk-content">
<h1 id="basic-loop">4. The basic conversational AI loop</h1>
<p>The basic "job to be done" of a voice AI agent is to listen to what a human says, respond in some useful way, then repeat that sequence.</p>
<p>Production voice agents today almost all have a very similar architecture. A voice agent program runs in the cloud and orchestrates the speech-to-speech loop. The agent program uses multiple AI models, some running locally to the agent, some accessed via APIs. The agent program also uses LLM function calling or structured outputs to integrate with back-end systems.</p>
<ol class="list-decimal">
<li>Speech is captured by a microphone on a user's device, encoded, and sent over the network to a voice agent program running in the cloud.</li>
<li>Input speech is transcribed, to create text input for the LLM.</li>
<li>Text is assembled into a context — a prompt — and inference is performed by an LLM. Inference output will often be filtered or transformed by the agent program logic.<sup>[5]</sup></li>
<li>Output text is sent to a text-to-speech model to create audio output.</li>
<li>Audio output is sent back to the user.</li>
</ol>
<p>You'll notice that the voice agent program is running in the cloud, and the text-to-speech, LLM, and speech-to-text processing are happening in the cloud. Over the long term, we expect to see more AI workloads running on-device. Today, though, <strong>production voice AI is very cloud-centric</strong>, for two reasons:</p>
<ol class="list-decimal">
<li>Voice AI agents need to use the best available AI models to reliably execute complex workflows at low latency. End-user devices do not yet have enough AI compute horsepower to run the best STT, LLM, and TTS models at acceptable latency.</li>
<li>The majority of commercial voice AI agents today are communicating with users via phone calls. For a phone call, there is no end-user device — at least, not one that you can run any code on!</li>
</ol>
<p>Let's dive<sup>[6]</sup> into this agent orchestration world and answer questions like:</p>
<ol class="list-decimal">
<li>What LLMs work best for voice AI agents?</li>
<li>How do you manage the conversation context during a long-running session?</li>
<li>How do you connect voice agents to existing back-end systems?<sup>[7]</sup></li>
<li>How do you know if your voice agents are performing well?</li>
</ol>
</div>
<div class="chunk-notes">
<div class="chapter-image">
<img src="images/Figure 0200.svg" class="image-hide-narrow" width="100%">
<p class="image-caption image-hide-narrow">The architecture of almost all production voice AI agents today</p>
</div>
<div class="chunk-footnotes">
<div class="footnote">
<p>[5] For example, to detect common LLM errors and safety issues.</p>
</div>
<div class="footnote">
<p>[6] Let's delve — ed.</p>
</div>
<div class="footnote">
<p>[7] For example, CRMs, proprietary knowledge bases, and call center systems.</p>
</div>
</div>
</div>
</div>
<div class="chunk-row">
<div class="chunk-content">
<h1 id="core-tech">5. Core technologies and best practices</h1>
</div>
</div>
<div class="chunk-row">
<div class="chunk-content">
<h2 id="latency">5.1. Latency</h2>
<p>Building voice agents is similar in most ways to other kinds of AI engineering. If you have experience building text-based, multi-turn AI agents, much of your experience from that domain will be useful in voice, as well.</p>
<p><strong>The big difference is latency.</strong></p>
<p>Humans expect fast responses in normal conversation. A response time of 500ms is typical. Long pauses feel unnatural.</p>
<p>It's worth learning how to accurately measure latency — from the end user's perspective — if you are building voice AI agents.</p>
<p>You will often see AI platforms quote latencies that are not true "voice-to-voice" measurements. This is generally not malicious. From the provider side of things, the easy way to measure latency is to measure inference time. So that's how providers get used to thinking about latency. However, this server-side view does not account for audio processing, phrase endpointing delay, network transport, and operating system overhead.</p>
<p><strong>Measuring voice-to-voice latency is easy to do manually.</strong></p>
<p>Simply record the conversation, load the recording into an audio editor, look at the audio waveform, and measure from the end of the user's speech to the beginning of the LLM's speech.</p>
<p>If you build conversational voice applications for production use, it's worthwhile to occasionally sanity check your latency numbers this way. Bonus points for adding simulated network packet loss and jitter when you do these tests!</p>
<p>Measuring true voice-to-voice latency is challenging to do programmatically. Some of the latency happens deep inside the operating system. So most observability tools just measure time-to-first-(audio)-byte. This is a reasonable proxy for total voice-to-voice latency, but again please note that things you don't measure — like phrase endpointing variation and network round-trip time — can become problematic if you have no way to track them.</p>
<p><strong>If you are building conversational AI applications, 1,500 ms voice-to-voice latency is an important target to aim for.</strong> Here's a breakdown of a voice-to-voice round trip from a user's microphone, to the cloud, and back. These numbers are fairly typical, and the total is about 1,200 ms. So 1,500 ms is challenging — if your telephony provider adds a few hundred milliseconds, you're over budget — but is possible to achieve with today's best AI models and an efficient framework like Pipecat.</p>
<table class="data-table latency-breakdown">
<thead>
<tr>
<th>Stage</th>
<th>Time (ms)</th>
</tr>
</thead>
<tbody>
<tr>
<td>macOS mic input</td>
<td>40</td>
</tr>
<tr>
<td>opus encoding</td>
<td>21</td>
</tr>
<tr class="network-row">
<td>network stacks and transit</td>
<td>10</td>
</tr>
<tr>
<td>packet handling</td>
<td>2</td>
</tr>
<tr>
<td>jitter buffer</td>
<td>40</td>
</tr>
<tr>
<td>opus decoding</td>
<td>1</td>
</tr>
<tr>
<td>transcription and endpointing</td>
<td>300</td>
</tr>
<tr>
<td>llm ttfb</td>
<td>650</td>
</tr>
<tr>
<td>sentence aggregation</td>
<td>20</td>
</tr>
<tr>
<td>tts ttfb</td>
<td>120</td>
</tr>
<tr>
<td>opus encoding</td>
<td>21</td>
</tr>
<tr>
<td>packet handling</td>
<td>2</td>
</tr>
<tr class="network-row">
<td>network stacks and transit</td>
<td>10</td>
</tr>
<tr>
<td>jitter buffer</td>
<td>40</td>
</tr>
<tr>
<td>opus decoding</td>
<td>1</td>
</tr>
<tr>
<td>macOS speaker output</td>
<td>15</td>
</tr>
<tr>
<td>Total ms</td>
<td>1293</td>
</tr>
</tbody>
</table>
<p class="table-caption">A voice-to-voice conversation round trip — latency breakdown.</p>
<p>We have demonstrated Pipecat agents that achieve voice-to-voice latency as low as 500 ms, by hosting all models within the same GPU-enabled cluster, and optimizing all models for latency instead of throughput.</p>
<p>Because latency is so important for voice use cases, latency will come up often throughout this guide.</p>
</div>
<div class="chunk-notes">
<div class="image-hide-narrow">
<img src="images/Figure 0300.svg" width="100%">
</div>
</div>
</div>
<div class="chunk-row">
<div class="chunk-content">
<h2 id="llms-for-voice">5.2. LLMs for voice use cases</h2>
<p>The release of GPT-4 in March 2023 kicked off the current era of voice AI. GPT-4 was the first LLM that could both sustain a flexible, multi-turn conversation and be prompted precisely enough to perform useful work.</p>
<p>Today, the most widely used models for voice agents are GPT-4.1, GPT-5.1, and Gemini 2.5 Flash.</p>
<p>These models combine:</p>
<ul class="arrow-list">
<li>Low latency.</li>
<li>Good instruction following.<sup>[8]</sup></li>
<li>Reliable function calling.<sup>[9]</sup></li>
<li>Low rates of hallucination and other kinds of inappropriate responses.</li>
<li>Dependable personality and tone.</li>
<li>Relatively low cost.</li>
</ul>
</div>
<div class="chunk-notes">
<div class="chunk-footnotes">
<div class="footnote">
<p>[8] How easy is it to prompt the model to do specific things?</p>
</div>
<div class="footnote">
<p>[9] Voice AI agents rely heavily on function calling.</p>
</div>
</div>
</div>
</div>
<div class="chunk-row">
<div class="chunk-content">
<p>Recently, several model families are challenging the GPT and Gemini model families on the Pareto frontier of intelligence and latency.</p>
<p>You'll notice that GPT-4.1 and Gemini 2.5 Flash are both relatively old models. Newer models in both families are "reasoning" models. They are much slower to produce content tokens. Gemini 3 models are particularly slow, so even though they perform well on intelligence benchmarks we generally can't use them for voice agents.</p>
<p>Our thinking has evolved since the earliest days of building voice agents, though. We spent a lot of time, in 2024, convincing people that agents could respond "as quickly as humans." We now have a lot of empirical evidence from deploying real-world voice agents for a wide range of use cases. People are happy talking to agents that respond within 1,500 ms.</p>
<p>It is critical, though, not to have latency spikes, or to let latency creep up above 1,500 ms as you add features like tool calling to your agents. And because latency is so important for voice use cases, latency will come up often throughout this guide.</p>
</div>
</div>
<div class="chunk-row">
<div class="chunk-content">
<table class="data-table model-comparison">
<thead>
<tr>
<th>Model</th>
<th>Median TTFT (ms)</th>
<th>P95 TTFT (ms)</th>
</tr>
</thead>
<tbody>
<tr>
<td>GPT-4.1</td>
<td>536</td>
<td>1771</td>
</tr>
<tr>
<td>Gemini 2.5 Flash</td>
<td>597</td>
<td>1137</td>
</tr>
<tr>
<td>Claude 4.5 Haiku</td>
<td>637</td>
<td>1615</td>
</tr>
<tr>
<td>GPT-5.1</td>
<td>739</td>
<td>1492</td>
</tr>
<tr>
<td>Nemotron 3 Ultra (self-hosted)</td>
<td>541</td>
<td>712</td>
</tr>
</tbody>
</table>
<p>A rough rule of thumb: LLM time-to-first-token of 600 ms or less is fast enough for most voice AI use cases.</p>
<p>Note that P95 times matter, too, and recently all providers have had worse P95 times from their public APIs than we would like to see.</p>
<p>It is possible, at scale, to have lower latency by purchasing committed inference compute from AWS, GCP, and Azure. Similarly, self-hosting open weights models (for example, NVIDIA Nemotron) becomes an interesting option if you have enough consistent usage to justify the baseline cost of always-on GPU instances.</p>
</div>
</div>
<div class="chunk-row">
<div class="chunk-content">
<h3 id="cost-comparison">5.2.1 Cost comparison</h3>
<p>Speaking of cost, per-token pricing has been dropping regularly and rapidly. And OpenAI, Google, and Anthropic all now support input token caching, which further reduces the cost for multi-turn conversations. (All voice agent conversations are multi-turn.)</p>
<table class="data-table model-comparison">
<thead>
<tr>
<th>Model</th>
<th>3-minute conversation</th>
<th>10-minute conversation</th>
<th>30-minute conversation</th>
</tr>
</thead>
<tbody>
<tr>
<td>Gemini 2.5 Flash</td>
<td>$0.002</td>
<td>$0.006</td>
<td>$0.024</td>
</tr>
<tr>
<td>Claude 4.5 Haiku</td>
<td>$0.006</td>
<td>$0.019</td>
<td>$0.075</td>
</tr>
<tr>
<td>GPT-5.1</td>
<td>$0.008</td>
<td>$0.025</td>
<td>$0.100</td>
</tr>
<tr>
<td>GPT-4.1</td>
<td>$0.019</td>
<td>$0.069</td>
<td>$0.318</td>
</tr>
</tbody>
</table>
<p class="table-caption">Approximate LLM cost for conversations of various lengths, in English. This assumes a 2,000 token system prompt.</p>
<p>See below for a discussion of estimating voice agent total costs (not just LLM costs).</p>
</div>
<div class="chunk-notes">
<div class="chunk-footnotes">
</div>
</div>
</div>
<div class="chunk-row">
<div class="chunk-content">
<h3 id="open-source">5.2.2 Open source / open weights</h3>
<p><strong>Voice AI use cases are demanding enough that it generally makes sense to use the best available model that can operate at real-time latency.</strong></p>
<p>This has limited the models that are used for voice AI to proprietary models.</p>
<p>However, open weights models are beginning to match closed models on important benchmarks. This is exciting, because open models expand what we can do with voice agents. Open models allow us to customize the inference stack (for example, prioritizing latency over throughput), fine-tune on our own data, and run the models on our own infrastructure.</p>
<p>Several promising open models have been released recently that we are beginning to use in voice agents and task subagents: the NVIDIA Nemotron 3 model family, Kimi 2.6, Gemma 4, and GLM 5.</p>
<p>See below for benchmark charts and Pareto frontier diagrams.</p>
</div>
<div class="chunk-notes">
<div class="chunk-footnotes">
<div class="footnote">
<p>[11] If you plan to fine-tune an LLM for your use case, an open weights model is a very good starting point. More on fine-tuning below.</p>
</div>
</div>
</div>
</div>
<div class="chunk-row">
<div class="chunk-content">
<h3 id="speech-to-speech">5.2.3 What about speech-to-speech models?</h3>
<p>A speech-to-speech LLM can be prompted with audio, rather than text, and can produce audio output directly. This eliminates the speech-to-text and text-to-speech parts of the voice agent orchestration loop.</p>
<p>The potential benefits of speech-to-speech models are:</p>
<ul class="arrow-list">
<li>Lower latency.</li>
<li>Improved ability to understand the nuances of human conversation.</li>
<li>More natural voice output.</li>
</ul>
<p>OpenAI, Google, and AWS all offer speech-to-speech models served via their APIs. NVIDIA and several other research labs have shipped speech-to-speech demos and research artifacts.</p>
<p>Speech-to-speech models do not follow instructions or call tools as reliably as text-mode LLMs. They are also slower, more expensive, less configurable, and harder to integrate into real-world agent systems.</p>
<p>On the other hand, today's best speech-to-speech models do sound very natural. OpenAI's gpt-realtime model really does sound like a preview of the voice AI future.</p>
</div>
</div>
<div class="chunk-row">
<div class="chunk-content">
<p>Here's how speech-to-speech models stack up against text-mode LLMs today:</p>
<ul class="arrow-list">
<li>Lower latency is possible in theory, for speech-to-speech models, but audio uses more tokens than text. Larger token contexts are slower for the LLM to process. In practice, today, both the OpenAI and Google speech-to-speech models are slower than a well-tuned cascaded (multi-model) voice agent.</li>
<li>Better understanding does seem to be a real benefit of these models. This is particularly apparent for Gemini 2.5 Flash and Gemini 3 Flash audio input.</li>
<li>Better natural voice output is clearly perceptible, today. In tests, most users rate output from the best speech-to-speech models as more natural than output from standalone text-to-speech models.</li>
</ul>
</div>
</div>
<div class="chunk-row">
<div class="chunk-content">
<ul class="arrow-list">
<li>The APIs for speech-to-speech models are significantly less flexible than they need to be for production voice agent engineering. Enterprise voice agents need to do a lot of context manipulation and summarization. This is possible with the OpenAI Real-time API, but because the API maintains its own internal version of the conversation context, the bookkeeping is tricky. Sophisticated context manipulation is not possible at all with the Gemini Live API, which is best regarded as an alpha release that is not ready for production.</li>
<li>Speech-to-speech APIs don't offer fast or reliable transcription. If you need accurate transcription of user input for compliance, downstream evals, or for use in an application UI, the transcription from the speech-to-speech APIs may not be good enough.</li>
<li>For use cases where input is mixed-language speech, the speech-to-speech models perform far, far better than a multi-model pipeline. Language learning applications, for example, greatly benefit from this strength of speech-to-speech models.</li>
</ul>
<p>It's also worth noting that speech-to-speech APIs are still relatively expensive. We built <a href="https://dub.sh/voice-agents-010" target="_blank">a calculator for the OpenAI Real-time API</a> that shows how cost scales with session length, factoring in OpenAI's very nice automatic token caching feature. An agent built with the OpenAI Real-time API is 3 to 5 times more expensive than an agent built with GPT-4.1.</p>
<p>We expect to see continued progress on the speech-to-speech front. But how quickly production voice AI applications will move from the multi-model approach to using speech-to-speech APIs is still an open question.</p>
</div>
<div class="chunk-notes">
<div class="chunk-footnotes">
<div class="footnote">
<p>[14] See <a href="https://latent.space/p/realtime-api" target="_blank">detailed notes about the Realtime API</a></p>
</div>
</div>
<div class="footnote">
<a href="https://dub.sh/voice-agents-010" target="_blank"><img src="images/Figure 0700 Spreadsheet.png" width="90%"></a>
<p class="image-caption">OpenAI Realtime API cost calculator</p>
</div>
</div>
</div>
<div class="chunk-row">
<div class="chunk-content">
<h2 id="benchmarks">5.3. LLM benchmarks</h2>
</div>
</div>
<div class="chunk-row">
<div class="chunk-content">
<h3 id="pareto-frontier-llm">5.3.1 The intelligence/latency Pareto frontier</h3>
<p>Intelligence, latency, and cost are all important factors when choosing which LLMs to use for our agents.</p>
<p>Because latency is particularly important for voice agents, we're often forced into a trade-off between using a model that is not as "smart" as we'd like, but has a fast TTFT.</p>
<p>This is particularly true because, for the past two years, the foundation labs have focused on reasoning models, which output "thinking" tokens before they produce actual content.</p>
<p>The user-visible response latency of a reasoning model isn't the time to first token. It's the time to first non-thinking token.</p>
<p>We maintain a <a href="https://github.com/kwindla/aiewf-eval" target="_blank">30-turn conversation benchmark</a> that tests LLM performance in a challenging, real-world voice agent scenario.</p>
<p>We've tracked two recent trends with great interest.</p>
<p>Starting in late 2025, new models started to saturate this benchmark. However, all these new models are too slow for voice use cases. We need a TTFT (first non-thinking token) faster than 700ms to use a model for realtime voice conversation. Still, saturating this hard benchmark is a big milestone.</p>
<p>This year, we've seen new performance jumps in open models. Nemotron 3 Ultra is the first open model that scores 100% on this benchmark. It's also the first model — open or proprietary — that both saturates the benchmark and delivers a TTFT below 700ms.</p>
<p>There are now two Pareto frontiers on the benchmark chart. Nemotron 3 Ultra (self-hosted), Kimi 2.6 (hosted on <a href="https://www.cerebras.ai/" target="_blank">Cerebras</a>) and Gemma 4 31b (hosted on <a href="https://getlilac.com/" target="_blank">Lilac</a>) define an open model Pareto frontier. GPT-4.1, Claude Haiku 4.5, and Claude Sonnet 5.6 form a separate proprietary model/first-party API frontier.</p>
<p>Open models outperform the closed models for latency-sensitive use cases.</p>
<p>The caveat here is that you have to self-host or commit significant inference volume to a new platform like Cerebras or Lilac. On the other hand, using an open model opens up other opportunities for fine-tuning and customization.</p>
<div class="chapter-image">
<img src="images/benchmark-text-table.png" alt="Voice agent benchmark leaderboard: pass rate, turn pass, tool use, instruction following, KB grounding, and TTFT for text-mode models" width="100%">
</div>
<div class="chapter-image">
<img src="images/benchmark-text-pareto.png" alt="Text models — latency vs. accuracy on the aiewf_medium_context benchmark, showing the open-weights and first-party-API Pareto frontiers" width="100%">
</div>
</div>
</div>
<div class="chunk-row">
<div class="chunk-content">
<h2 id="speech-to-text">5.4. Speech-to-text</h2>
<p>Speech-to-text is the "input" stage for voice AI. Speech-to-text is also commonly referred to as <i>transcription</i> or <i>ASR</i> (automatic speech recognition).</p>
<p>For voice AI use cases, we need very low transcription latency and very low word error rate.</p>
<p>Today there are a number of transcription models that deliver excellent accuracy at very low latency.</p>
</div>
<div class="chunk-notes">
</div>
</div>
<div class="chunk-row">
<div class="chunk-content">
<h3 id="pareto-frontier">5.4.1 The STT Pareto frontier</h3>
<p>We maintain speech-to-text benchmarks that test models in configurations and on data from real-world voice agents:</p>
<p><a href="https://github.com/pipecat-ai/stt-benchmark/" target="_blank">https://github.com/pipecat-ai/stt-benchmark/</a></p>
<p>On the chart, you can see a Pareto frontier cluster with models from NVIDIA, Deepgram, Soniox, Cartesia, AssemblyAI, and Speechmatics all performing extremely well.</p>
<p>Deepgram is the long-time leader in real-time transcription. Soniox offers low-latency transcription models across a wide range of languages. Speechmatics is a UK company that trains speech models for real-time, broadcast, and on-device applications. AssemblyAI offers innovative features such as multi-turn context carry-over. Cartesia's text-to-speech models have won significant market share in the voice generation market and they are now shipping competitive speech-to-text models as well. NVIDIA's model is completely open source.</p>
<p>All of these models are available through APIs or as Docker containers that customers can run on their own infrastructure. Deepgram's models are also available through AWS SageMaker.</p>
<div class="chapter-image">
<img src="images/stt_pareto_frontier_p95.png" alt="STT Pareto Frontier: TTFS P95 Latency vs Accuracy" width="100%">
<p class="image-caption">Pareto Frontier Services (TTFS P95)</p>
</div>
<p>Most people start out using speech-to-text models via an API. Managing a scalable GPU cluster is a significant ongoing devops job to take on, so moving from an API to hosting models on your own infrastructure is not something you should do without a good reason. But there definitely <em>are</em> good reasons for self-hosting, which include:</p>
<ul class="arrow-list">
<li>Keeping audio and transcription data private. Your company policies may prohibit sending user data outside your own infrastructure. You may have legal requirements to process data only in a specific geographic region.</li>
<li>Reducing latency. API providers may not have inference servers in regions where you have users. This is a significant issue for voice agents. From Europe, round-trip network time to the US is ~250 ms; from India, ~350 ms.</li>
</ul>
<p>Several providers also offer fine-tuning services, which can help lower error rates if your use case includes relatively unusual vocabularies, speech styles, or accents.</p>
<p>Fine-tuning on your own data is also a good reason to explore using the NVIDIA open models.</p>
</div>
</div>
<div class="chunk-row">
<div class="chunk-content">
<h3 id="prompting-help">5.4.2 Prompting can help the LLM.</h3>
<p>A large percentage of transcription errors result from the very small amount of context that the transcription model has available in a realtime stream.</p>
<p>Today's LLMs are smart enough to work around transcription errors. When the LLM is performing inference it has access to the full conversation context. So you can tell the LLM that the input is a transcription of user speech, and that it should reason accordingly.</p>
<pre><code>You are a helpful, concise, and reliable voice assistant. Your primary goal is to understand the user's spoken requests, even if the speech-to-text transcription contains errors. Your responses will be converted to speech using a text-to-speech system. Therefore, your output must be plain, unformatted text.
When you receive a transcribed user request:
1. Silently correct for likely transcription errors. Focus on the intended meaning, not the literal text. If a word sounds like another word in the given context, infer and correct. For example, if the transcription says "buy milk two tomorrow" interpret this as "buy milk tomorrow".
2. Provide short, direct answers unless the user explicitly asks for a more detailed response. For example, if the user says "what time is it?" you should respond with "It is 2:38 AM". If the user asks "Tell me a joke", you should provide a short joke.
3. Always prioritize clarity and accuracy. Respond in plain text, without any formatting, bullet points, or extra conversational filler.
4. If you are asked a question that is time dependent, use the current date, which is February 3, 2025, to provide the most up to date information.
5. If you do not understand the user request, respond with "I'm sorry, I didn't understand that."
Your output will be directly converted to speech, so your response should be natural-sounding and appropriate for a spoken conversation.
</code></pre>
<p class="image-caption">Example prompt language for a voice AI agent.</p>
</div>
</div>
<div class="chunk-row">
<div class="chunk-content">
<h2 id="text-to-speech">5.5. Text-to-speech</h2>
<p>Text-to-speech is the output stage of the voice-to-voice processing loop.</p>
<p>Voice AI developers choose a voice model/service based on:</p>
<ul class="arrow-list">
<li>How natural the voices sound (overall quality)<sup>[16]</sup></li>
<li>Latency<sup>[17]</sup></li>
<li>Cost</li>
<li>Language support</li>
<li>Word-level timestamp support</li>
<li>Ability to customize voices, accents, and pronunciations</li>
</ul>
</div>
<div class="chunk-notes">
<div class="chunk-footnotes">
<div class="footnote">
<p>[16] Pronunciation, intonation, pacing, stress, rhythm, emotional valence.</p>
</div>
<div class="footnote">
<p>[17] Time to first audio byte.</p>
</div>
</div>
</div>
</div>
<div class="chunk-row">
<div class="chunk-content">
<p>Voice options expanded markedly in 2024 and 2025. New startups appeared on the scene. Best-in-class voice quality went way up. And every provider improved latency.</p>
<p>As is the case for speech-to-text, all of the big cloud providers have text-to-speech products.<sup>[18]</sup> But most voice AI developers are not using them, because models from startups are currently better.</p>
</div>
<div class="chunk-notes">
<div class="chunk-footnotes">
<div class="footnote">
<p>[18] Azure AI Speech, Amazon Polly, and Google Cloud Text-to-Speech.</p>
</div>
</div>
</div>
</div>
<div class="chunk-row">
<div class="chunk-content">
<p>The labs that have the most traction for realtime conversational voice models are (in alphabetical order):</p>
<ul class="arrow-list">
<li>Cartesia – Uses an innovative state-space model architecture.</li>
<li>Deepgram – Prioritizes latency and low cost. Deepgram's transcription models were the original low-latency/high-accuracy ML models for voice AI.</li>
<li>ElevenLabs – Emphasizes emotional and contextual realism.</li>
<li>Gradium – A commercial spin-out of the non-profit French lab, Kyutai, which has produced some of the world's most innovative speech model work.</li>
<li>Inworld – Roots in AI technology innovations for video games.</li>
</ul>
<p>All four companies have strong models, experienced engineering teams, and stable and performant APIs. The Cartesia, Deepgram, and Gradium models can be deployed on your own infrastructure.</p>
<table class="data-table voice-model-breakdown">
<thead>
<tr>
<th> </th>
<th>Cost per minute (approx)</th>
<th>Median TTFA (ms)</th>
<th>P95 TTFA (ms)</th>
</tr>
</thead>
<tbody>
<tr>
<td>Cartesia Sonic 3.5</td>
<td>$0.028</td>
<td>195</td>
<td>240</td>
</tr>
<tr>
<td>Deepgram Aura-2</td>
<td>$0.024</td>
<td>310</td>
<td>600</td>
</tr>
<tr>
<td>ElevenLabs Turbo v2.5</td>
<td>$0.050</td>
<td>330</td>
<td>670</td>
</tr>
<tr>
<td>Gradium</td>
<td>$0.032</td>
<td>235</td>
<td>320</td>
</tr>
<tr>
<td>Inworld TTS 1.5 Max</td>
<td>$0.009</td>
<td>337</td>
<td>560</td>
</tr>
</tbody>
</table>
<p class="table-caption">Approximate cost per minute (at scale) and time-to-first-audio metrics – June 2025. Note that cost depends on committed volume and features used.</p>
<p>As with speech-to-text, there is wide variance in quality and support for non-English voice models. If you are building voice AI for non-English use cases, you will likely need to do more extensive testing — test more services and more voices to find a solution that you are happy with.</p>
<p>All voice models will mispronounce words some of the time, and will not necessarily know how to pronounce proper nouns or unusual words.</p>
<p>Some services offer the ability to steer pronunciation. This is helpful if you know in advance that your text output will include specific proper nouns. If your voice service does not support phonetic steering, you can prompt your LLM to output "sounds-like" spellings of specific words. For example, in-vidia instead of NVIDIA.</p>
<pre ><code class="nobreak">
Replace "NVIDIA" with "in vidia" and replace <br/>
"GPU" with "gee pee you" in your responses.
</code></pre>
<p class="image-caption">Example prompt language to steer pronunciation via LLM text output</p>
<p>For conversational voice use cases, being able to track what text the user heard is important for maintaining accurate conversation context. This requires that a model generate word-level timestamp metadata in addition to the audio, and that the timestamp data be reconstructible backwards to the original input text. This is a relatively new capability for voice models. All of the models in the table above except ElevenLabs Flash support word-level timestamps.</p>
<pre><code>
{
"type": "timestamps",
"context_id": "test-01",
"status_code": 206,
"done": false,
"word_timestamps": {
"words": ["What's", "the", "capital", "of", "France?"],
"start": [0.02, 0.3, 0.48, 0.6, 0.8],
"end": [0.3, 0.36, 0.6, 0.8, 1]
}
}
</code></pre>
<p class="image-caption">Word-level timestamps from the Cartesia API.</p>
<p>In addition, a really solid realtime streaming API is helpful. Conversational voice applications often trigger multiple audio inferences in parallel. Voice agent code needs to be able to interrupt in-progress inference and to correlate each inference request to one output stream. Streaming APIs from voice model providers are all relatively new and still evolving.</p>
<p>We expect voice model progress to continue in 2026.</p>
</div>
<div class="chunk-notes">
<div class="chunk-footnotes">
</div>
</div>
</div>
<div class="chunk-row">
<div class="chunk-content">
<h2 id="audio-processing">5.6. Audio processing</h2>