Describe the bug
A single RTP frame that the decoder rejects tears down an entire bridged call, killing both legs.
When switch_core_session_write_frame() has to transcode — an Opus leg bridged to a G.711 leg, for instance — the decode happens on the write path. Any decode failure falls into the default: case of the status switch, which logs Codec %s decoder error! and then goto error with the fatal status still set:
https://github.com/signalwire/freeswitch/blob/c1bb5c6ab3/src/switch_core_media.c#L16061-L16063
switch_ivr_bridge() treats every non-SWITCH_STATUS_SUCCESS write as terminal, so it leaves the bridge loop and both legs are hung up:
https://github.com/signalwire/freeswitch/blob/c1bb5c6ab3/src/switch_ivr_bridge.c#L818-L823
The resulting hangup cause is NORMAL_CLEARING on both legs, so nothing in the CDR or hangup-cause reporting points at the decoder. The only trace is the decoder error! log line 20 ms earlier.
The read path does not behave this way. switch_core_io.c logs the same decode error, drops the frame and keeps the call up:
https://github.com/signalwire/freeswitch/blob/c1bb5c6ab3/src/switch_core_io.c#L633-L643
The write path is the only decode site that dies on the first bad frame. It even already contains the precedent for the correct behaviour a few lines above: case SWITCH_STATUS_BREAK: sets SWITCH_STATUS_SUCCESS and jumps to the same error label, i.e. drops the frame without failing the write.
To Reproduce
- Bridge an Opus leg to a PCMA or PCMU leg, so
switch_core_session_write_frame() must transcode. (Any transcoding bridge works; Opus is just the easiest codec to feed an invalid payload to. An armed jitter buffer — e.g. rtp_jitter_buffer_during_bridge=true — makes it much easier to hit in the wild, but is not required.)
- Inject one RTP packet on the Opus leg whose payload is an Opus framing violation. Two bytes are enough:
0xFF 0x00 — a code-3 TOC with a frame count of 0, which opus_decode() rejects with OPUS_INVALID_PACKET. Any RTP injector (sipp with a crafted pcap, a patched sender, tcpreplay) will do; a real network only has to corrupt a payload once.
- The call ends within one packet interval (~20 ms).
Expected behavior
The undecodable frame is dropped, a gap of one packet interval is heard, and the call continues — the same thing that happens when the identical packet arrives on the read path.
Actual behavior
Both legs are hung up with NORMAL_CLEARING.
Package version or git hash
Reproduced on 1.11.1; the code is unchanged on master as of c1bb5c6ab3.
Trace logs
[ERR] mod_opus.c:931 Decoder Error: corrupted stream fs:960 plc:false!
[ERR] switch_core_media.c:16061 Codec OPUS (STANDARD) decoder error!
[DEBUG] switch_ivr_bridge.c:819 <b-leg> ending bridge by request from write function
[NOTICE] switch_ivr_bridge.c:... Hangup <a-leg> [CS_EXECUTE] [NORMAL_CLEARING]
[NOTICE] switch_core_session.c:... Hangup <b-leg> [CS_EXCHANGE_MEDIA] [NORMAL_CLEARING]
backtrace from core file
N/A — no crash, the call is hung up cleanly.
Prior history
This is not new, and it has been hit before:
Impact
In a production deployment carrying Opus↔G.711 transcoding bridges this accounts for roughly 0.5% of answered bridged calls per day being dropped mid-conversation, with no diagnosable hangup cause.
I have a fix for this and will open a PR shortly.
Describe the bug
A single RTP frame that the decoder rejects tears down an entire bridged call, killing both legs.
When
switch_core_session_write_frame()has to transcode — an Opus leg bridged to a G.711 leg, for instance — the decode happens on the write path. Any decode failure falls into thedefault:case of the status switch, which logsCodec %s decoder error!and thengoto errorwith the fatal status still set:https://github.com/signalwire/freeswitch/blob/c1bb5c6ab3/src/switch_core_media.c#L16061-L16063
switch_ivr_bridge()treats every non-SWITCH_STATUS_SUCCESSwrite as terminal, so it leaves the bridge loop and both legs are hung up:https://github.com/signalwire/freeswitch/blob/c1bb5c6ab3/src/switch_ivr_bridge.c#L818-L823
The resulting hangup cause is
NORMAL_CLEARINGon both legs, so nothing in the CDR or hangup-cause reporting points at the decoder. The only trace is thedecoder error!log line 20 ms earlier.The read path does not behave this way.
switch_core_io.clogs the same decode error, drops the frame and keeps the call up:https://github.com/signalwire/freeswitch/blob/c1bb5c6ab3/src/switch_core_io.c#L633-L643
The write path is the only decode site that dies on the first bad frame. It even already contains the precedent for the correct behaviour a few lines above:
case SWITCH_STATUS_BREAK:setsSWITCH_STATUS_SUCCESSand jumps to the sameerrorlabel, i.e. drops the frame without failing the write.To Reproduce
switch_core_session_write_frame()must transcode. (Any transcoding bridge works; Opus is just the easiest codec to feed an invalid payload to. An armed jitter buffer — e.g.rtp_jitter_buffer_during_bridge=true— makes it much easier to hit in the wild, but is not required.)0xFF 0x00— a code-3 TOC with a frame count of 0, whichopus_decode()rejects withOPUS_INVALID_PACKET. Any RTP injector (sipp with a crafted pcap, a patched sender,tcpreplay) will do; a real network only has to corrupt a payload once.Expected behavior
The undecodable frame is dropped, a gap of one packet interval is heard, and the call continues — the same thing that happens when the identical packet arrives on the read path.
Actual behavior
Both legs are hung up with
NORMAL_CLEARING.Package version or git hash
Reproduced on 1.11.1; the code is unchanged on master as of
c1bb5c6ab3.Trace logs
backtrace from core file
N/A — no crash, the call is hung up cleanly.
Prior history
This is not new, and it has been hit before:
mod_opusby returningSWITCH_STATUS_NOOPfromswitch_opus_decode(), explicitly because "this makes switch_core_media hangup the call".NOOPdoes not mean "error" — which re-exposed the core behaviour that [mod_opus] Do not hangup call on decode error - fix 86a5ee3509 #982 had been papering over. The review discussion on both PRs agrees that hanging up on a decode error is wrong; the disagreement was only about which status to return, i.e. it was being fixed at the codec layer rather than at the site that decides to tear the call down.Impact
In a production deployment carrying Opus↔G.711 transcoding bridges this accounts for roughly 0.5% of answered bridged calls per day being dropped mid-conversation, with no diagnosable hangup cause.
I have a fix for this and will open a PR shortly.