[Core] Do not hang up the call on a write-path decoder error - #3103
Open
yosif111 wants to merge 1 commit into
Open
[Core] Do not hang up the call on a write-path decoder error#3103yosif111 wants to merge 1 commit into
yosif111 wants to merge 1 commit into
Conversation
A single RTP frame that the decoder rejects currently tears down an entire bridged call. When switch_core_session_write_frame() has to transcode -- an Opus leg bridged to a G.711 leg, for instance -- the decode happens on the write path, and any failure falls into the default case of the status switch, which logs "Codec %s decoder error!" and jumps to the error label with the fatal status still set. switch_ivr_bridge() treats every non-SUCCESS write as terminal, so it leaves the bridge loop and hangs up both legs. The resulting hangup cause is NORMAL_CLEARING on both sides, so nothing in the CDR points at the decoder and the failure is very hard to attribute. The read path does not behave this way: switch_core_io.c logs the same error, drops the frame and keeps the call up. The write path is the only decode site that dies on the first bad frame, and it already contains the precedent for the right behaviour a few lines above, where SWITCH_STATUS_BREAK sets SWITCH_STATUS_SUCCESS and jumps to the same label, dropping the frame without failing the write. Mirror the read path. Count consecutive write-path decode failures in a new session->write_decoder_errors and return SWITCH_STATUS_SUCCESS for the first nine, so an undecodable frame is simply dropped and the call survives with a one-packet gap. The tenth consecutive failure propagates as before, so a genuinely broken stream still terminates. The counter is reset after any successful decode, which keeps isolated corrupt packets from ever accumulating. It is separate from session->decoder_errors because the read and write paths use different codecs and run on different threads. This has been reported before. mod_opus worked around it in signalwire#982 by returning SWITCH_STATUS_NOOP from switch_opus_decode(), which was correctly reverted in signalwire#2381 since NOOP does not mean "error"; fixing the codec status re-exposed the core behaviour underneath it. The same symptom was reported on freeswitch-users in 2014 with no fix posted: https://lists.freeswitch.org/pipermail/freeswitch-users/2014-November/109328.html Fixes signalwire#3102 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
One RTP frame that the decoder rejects currently kills an entire bridged call — both legs.
When
switch_core_session_write_frame()has to transcode (an Opus leg bridged to a G.711 leg, say), the decode happens on the write path. Any decode failure lands in thedefault:case of the status switch, which logsCodec %s decoder error!and thengoto errorwith the fatal status still set.switch_ivr_bridge()treats every non-SWITCH_STATUS_SUCCESSwrite as terminal, leaves the bridge loop, and both legs are hung up — withNORMAL_CLEARING, so nothing in the CDR or hangup-cause reporting points at the decoder.The read path already does the right thing.
switch_core_io.clogs the same error, drops the frame and keeps the call up, using asession->decoder_errorscounter and a limit of 10. The write path is the only decode site that dies on the first bad frame.It also already contains the precedent for the correct behaviour a few lines above:
case SWITCH_STATUS_BREAK:setsSWITCH_STATUS_SUCCESSand jumps to the sameerrorlabel — i.e. drop the frame, don't fail the write. This change makes an isolated decode error behave the same way.The fix adds
session->write_decoder_errors, returnsSWITCH_STATUS_SUCCESSfor the first nine consecutive failures so the undecodable frame is simply dropped, and lets the tenth propagate as before so a genuinely broken stream still terminates. The counter resets after any successful decode. It is a separate field fromdecoder_errorsbecause the read and write paths use different codecs and run on different threads.Prior history
This is a long-standing issue and the project has been round it once already:
mod_opusby returningSWITCH_STATUS_NOOPfromswitch_opus_decode(), explicitly because "this makes switch_core_media hangup the call".NOOPdoes not mean "error" — which re-exposed the core behaviour [mod_opus] Do not hangup call on decode error - fix 86a5ee3509 #982 had been papering over.The review discussion on both of those PRs agrees that hanging up on a decode error is wrong; the disagreement was only over which status the codec should return. This PR fixes it at the site that actually decides to tear the call down, so no codec has to lie about its status to keep a call alive.
Type of Change
Related Issues
Fixes #3102
Relates to #982, #2381
Testing
Repro recipe (enough to reproduce with sipp or any RTP injector):
switch_core_session_write_frame()must transcode. An armed jitter buffer (rtp_jitter_buffer_during_bridge=true) makes this much easier to hit in the wild but is not required.0xFF 0x00— a code-3 TOC with a frame count of 0, whichopus_decode()rejects withOPUS_INVALID_PACKET.A/B tested on 1.11.1 with two builds differing only by this hunk:
NORMAL_CLEARINGdecoder error!log, frame dropped, call continues with a single 20 ms gapBuild verification: the branch is built against master (
c1bb5c6) —./bootstrap.sh -j && ./configure && make libfreeswitch.laon Debian bookworm/amd64.Field data: this has been running in a production deployment carrying Opus↔G.711 transcoding bridges. Before the fix it accounted for roughly 0.5% of answered bridged calls per day being dropped mid-conversation with no diagnosable hangup cause; after the fix that class of drop is gone and the
decoder error!log line still appears at the same rate, so the bad frames are being dropped rather than hidden.Checklist
Additional Notes
One deliberate difference from
switch_core_io.c. On the read path thesession->decoder_errors = 0;reset sits immediately after the switch, on the same fall-through the tolerated-error branch takes — so that counter is zeroed again on every tolerated error and never actually reaches the limit. Here the error branch ends ingoto error, which skips the reset, sowrite_decoder_errorsreally does count consecutive failures and the give-up path is reachable. Happy to change it to matchswitch_core_io.cbyte-for-byte instead if you'd rather have exact symmetry, or to send a follow-up fixing the read-path reset placement — just say which you prefer.On the threshold. 10 is taken from the read path rather than chosen; it is a fixed constant there too. If you'd prefer it configurable (a global or a channel variable) I'm glad to add that.
On a unit test. I didn't add one because there is no existing harness that exercises
switch_core_session_write_frame()with a bridged pair;tests/unit/switch_core_codec.c(added in #2381) coversswitch_opus_decode()returningSWITCH_STATUS_FALSE, which is the input to this path but not the behaviour being fixed. Happy to add one if you can point me at the right place to hang it.🤖 Generated with Claude Code