Skip to content

Commit a595302

Browse files
committed
feat: Implement transport-wide congestion control (TWCC) feedback mechanism
- Added `transportWideCongestionControl` method to `RTCPFeedbackPolicy` for generating TWCC feedback packets. - Introduced `RTPHeaderExtension` struct to handle one-byte header extensions in RTP packets. - Enhanced `RTPPacket` to support header extensions and modified encoding/decoding logic accordingly. - Updated `SubscribeRTCPFeedbackPlanner` to include TWCC feedback in response to signals. - Modified `SubscriberMediaReceivePipeline` to track TWCC observations and build feedback packets. - Added tests for TWCC feedback encoding/decoding, planner behavior, and integration with RTP packets. - Updated documentation to reflect new TWCC features and improvements in feedback handling.
1 parent 4d4668f commit a595302

20 files changed

Lines changed: 1013 additions & 27 deletions

README.md

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -113,13 +113,17 @@ bitrate/FPS caps to VideoToolbox. Subscriber-side recommendations can also be
113113
planned and sent as LiveKit `UpdateTrackSettings` requests, while observed
114114
subscriber RTP/Sender Report state can generate scheduled RTCP Receiver
115115
Reports with DLSR timing and REMB bitrate feedback over the secure subscriber
116-
RTCP transport. `RoomOptions` can opt into automatically applying subscriber
117-
adaptive track settings from those estimates, and can set default LiveKit
116+
RTCP transport, and the RTCP layer can encode/decode TWCC feedback with
117+
bounded arrival-time feedback planning. The subscriber RTP path can parse
118+
one-byte RTP header extensions, observe the default TWCC transport sequence
119+
extension, preserve encrypted SRTCP feedback bytes until decrypt, and dispatch
120+
bounded TWCC feedback over the secure subscriber RTCP transport. `RoomOptions`
121+
can opt into automatically applying subscriber adaptive track settings from those estimates, and can set default LiveKit
118122
`adaptive_stream`, subscriber pause, and data-track auto-subscribe signaling
119123
preferences, with per-connection `ConnectOptions` overrides.
120124
Those are still only building blocks until they are expanded into complete
121-
congestion control, live-validated subscriber policy, weak-network recovery, and
122-
end-to-end LiveKit validation.
125+
live TWCC/REMB interop or equivalent congestion control, live-validated
126+
subscriber policy, weak-network recovery, and end-to-end LiveKit validation.
123127

124128
Those requirements are intentionally production blockers. A release can only be
125129
called production-ready after automated LiveKit integration tests, weak-network
@@ -401,7 +405,8 @@ recommendations, VideoToolbox bitrate/FPS recommendation application, camera
401405
publish backpressure/drop control, subscriber adaptive track-settings planning,
402406
manual subscriber video-quality preset signaling, publisher active video-layer
403407
`UpdateVideoLayers` signaling, and opt-in automatic signaling, subscriber Receiver Report
404-
generation/cadence/sending, REMB packet/planner/sending, opt-in subscriber
408+
generation/cadence/sending, REMB packet/planner/sending, RTP header-extension
409+
parsing, TWCC observation/packet/planner/send-loop primitives, opt-in subscriber
405410
Opus playout scheduling, opt-in subscriber H.264 VideoToolbox decode-to-pixel-buffer
406411
scheduling, public `SubscriberVideoFrameRenderer` handoff, UIKit/AppKit
407412
`VideoView` sample-buffer display, opt-in voice-chat audio-session

Sources/LiveKitNative/Core/ProductionReadiness.swift

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ public extension LiveKitNative {
3535
"SCTP data channel support includes packet-level DCEP/DataPacket planning, manager-assigned stream routing, queued local publish flushing, inbound DataPacket-to-RoomEvent plumbing, publisher SDP data-channel negotiation, subscriber-side data-channel receive-loop plumbing, OpenSSL DTLS application-data transport coverage, default Room shared-media DTLS/SRTP demux binding, deterministic packet fragmentation/reassembly envelopes, fragmented-packet retransmission scheduling on the DTLS-backed packet transport, opt-in standards-shaped SCTP association bootstrap with DATA/SACK exchange, fragmented DATA reassembly, duplicate DATA TSN suppression, contiguous cumulative SACK tracking with gap/duplicate reporting, ordered contiguous TSN delivery, outbound DATA TSN acknowledgement tracking with SACK gap-triggered retransmission of unacknowledged outbound DATA TSN holes, deterministic cwnd/rwnd send gating with queued DATA flush after SACK, slow-start window growth, gap-SACK fast-recovery window reduction, recovery reset that reopens LiveKit data channels after association restart, and default-gated two-client LiveKit DataPacket publish/receive smoke over that standards-shaped association transport, but full standards-compliant SCTP association state, complete RFC congestion control, LiveKit-validated data-channel recovery, and E2E hardening are incomplete.",
3636
"Signal reconnect resets stale peer negotiation state, restarts local ICE credentials, rebuilds retained subscriber answer and publisher offer SDP with fresh ICE credentials, sends SyncState for retained subscription/local media/data publication state, re-sends local ICE trickle/final-trickle when media startup is configured, preserves publisher offer track state for later publish/unpublish re-offers, clears stale local media/data publication state for server/SFU unpublish responses, and resets injected publisher data channels for post-reconnect DCEP reopen in unit tests, but live media recovery and LiveKit integration hardening are incomplete.",
3737
"General production video meetings require the new audio-session primitives to be validated with meeting-grade capture/playout, echo cancellation, route changes, Bluetooth behavior, interruptions, background/foreground handling, and timing on real iOS devices.",
38-
"A default subscriber RTP receive pipeline now integrates jitter buffering with bounded NACK/PLI feedback, scheduled RTCP Receiver Reports with DLSR timing from observed RTP/Sender Report state, REMB bitrate-feedback packet planning/sending, deterministic RTCP receiver-report bandwidth estimation, adaptive video quality recommendations, publisher RTCP receiver-report ingestion, H.264 encoder bitrate/FPS recommendation application, manual subscriber quality preset signaling, opt-in automatic subscriber adaptive track-settings dispatch, publisher active video-layer UpdateVideoLayers signaling, single-layer H.264 AddTrack high-quality metadata, and camera publish frame backpressure primitives, but general production video meetings still require full packet-loss recovery, TWCC/full REMB interop or equivalent congestion control, default-on LiveKit-validated subscriber adaptation policy, complete encoder control policy, actual multi-encoder simulcast/SVC media production, and weak-network E2E validation.",
38+
"A default subscriber RTP receive pipeline now integrates jitter buffering with bounded NACK/PLI feedback, scheduled RTCP Receiver Reports with DLSR timing from observed RTP/Sender Report state, REMB bitrate-feedback packet planning/sending, RTP one-byte header-extension parsing, TWCC feedback packet encode/decode with bounded arrival-time feedback planning, default subscriber TWCC observation and protected RTCP feedback dispatch, deterministic RTCP receiver-report bandwidth estimation, adaptive video quality recommendations, publisher RTCP receiver-report ingestion, H.264 encoder bitrate/FPS recommendation application, manual subscriber quality preset signaling, opt-in automatic subscriber adaptive track-settings dispatch, publisher active video-layer UpdateVideoLayers signaling, single-layer H.264 AddTrack high-quality metadata, and camera publish frame backpressure primitives, but general production video meetings still require full packet-loss recovery, live TWCC/REMB interop or equivalent congestion control, default-on LiveKit-validated subscriber adaptation policy, complete encoder control policy, actual multi-encoder simulcast/SVC media production, and weak-network E2E validation.",
3939
"General production video meetings require automated multi-participant, weak-network, TURN-only, reconnect, long-running soak, battery, and thermal tests on real devices.",
4040
"End-to-end LiveKit server compatibility tests are not automated in CI."
4141
],

Sources/LiveKitNative/LiveKitNative.docc/LiveKitNative.md

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -125,8 +125,12 @@ low/medium/high/off reception, and apps can use public subscriber
125125
video-quality presets without hand-coding dimensions or FPS caps. Observed
126126
subscriber RTP/Sender Report state can produce scheduled RTCP Receiver Reports
127127
with DLSR timing plus REMB bitrate feedback over the subscriber secure RTCP
128-
transport. `RoomOptions` can opt into deduplicated automatic subscriber
129-
`UpdateTrackSettings` dispatch from the current receiver-report estimate, and
128+
transport, and the RTCP layer includes TWCC feedback encode/decode plus
129+
bounded arrival-time feedback planning. The subscriber RTP path parses one-byte
130+
RTP header extensions, observes the default TWCC transport sequence extension,
131+
preserves encrypted SRTCP feedback bytes until decrypt, and can send bounded
132+
TWCC feedback over the subscriber secure RTCP transport. `RoomOptions` can opt
133+
into deduplicated automatic subscriber `UpdateTrackSettings` dispatch from the current receiver-report estimate, and
130134
publisher active video-layer `UpdateVideoLayers` signaling is available for
131135
Dynacast-style layer availability updates once real multi-layer media is
132136
validated.

Sources/LiveKitNativeWebRTC/MediaQualityAdaptation.swift

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -266,7 +266,9 @@ package final class RTCPBandwidthEstimateStore: @unchecked Sendable {
266266
case .pictureLossIndication,
267267
.transportLayerNACK,
268268
.receiverEstimatedMaximumBitrate,
269-
.applicationLayerFeedback:
269+
.transportWideCongestionControl,
270+
.applicationLayerFeedback,
271+
.raw:
270272
reports = []
271273
}
272274

0 commit comments

Comments
 (0)