TELCORE-355: propagate concrete recorder open-error detail on record_session_error - #638
Draft
minhtuan1407-telnyx wants to merge 2 commits into
Conversation
added 2 commits
August 11, 2026 16:18
…session_error When a record_session open fails (e.g. the recorder-server RTMP timeout in the 2026-08-07 incident), the record_session_error CUSTOM event only carried Record-Error="Error opening file", with the specific cause (Connection timed out vs connection refused/DNS/other) visible only in the mod_av log line. This made it impossible for downstream metrics/alerting (mod_telnyx) to distinguish a recorder-server timeout from any other open failure. - mod_av (avformat.c): on avio_open2/avio_open failure, stash the concrete error text and an explicit rw_timeout flag on the otherwise-unused handle->event (Record-Open-Error-Detail, Record-Open-Timeout). The ETIMEDOUT check is exact (AVERROR(ETIMEDOUT)), not string matching. - switch_ivr_async.c: send_record_error_event() forwards those as Record-Error-Detail / Record-Open-Timeout headers on the event when the file handle carries them; the three record_session open-failure call sites pass the file handle and destroy handle->event after use. Headers are absent for file modules that don't populate them (backwards compatible).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Propagate the concrete recorder open-error (and an explicit timeout flag) on the
record_session_errorevent, so downstream metrics/alerting can distinguish a recorder-server timeout from any other recording open failure.Linear: https://linear.app/telnyx/issue/TELCORE-355/add-b2bua-alerting-for-recorder-server-timeouts
Why
In the 2026-08-07 incident (~170k calls affected), recorder-server RTMP timeouts manifested as:
The
record_session_errorCUSTOM event fired, but it only carriedRecord-Error = "Error opening file"— the specific cause ("Connection timed out" vs connection-refused / DNS / other) lived only in themod_av/avformat.clog line, never in the event. Somod_telnyx(the Prometheus scrape endpoint) could not build a timeout-specific counter/alert without a core change.Change
mod_av(avformat.c): onavio_open2/avio_openfailure, stash the concrete error text and an explicit timeout flag on the otherwise-unusedhandle->event:Record-Open-Error-Detail=av_strerrortextRecord-Open-Timeout=trueiffret == AVERROR(ETIMEDOUT)(exact code check — not string matching;ETIMEDOUTis precisely how therw_timeoutinterrupt callback surfaces).switch_ivr_async.c:send_record_error_event()gains aswitch_file_handle_t *fhparam and forwards those asRecord-Error-Detail/Record-Open-Timeoutheaders when present. The threerecord_sessionopen-failure call sites pass the handle andswitch_event_destroy(&fh->event)after use.Headers are absent for file modules that don't populate them (WAV, sndfile, etc.) → fully backwards compatible; existing consumers see the same
Record-Errorthey always did.Lifetime / safety notes
handle->eventis unused on the recording open path today (git grepconfirms no reader inswitch_ivr_async.crecord_session). It is aSWITCH_EVENT_CLONEevent with its own allocation, not tied tofh->memory_pool— so it survivesswitch_core_file_open'sfail:cleanup (which only destroysfh->paramsand, when it owns it,fh->memory_pool).record_helper_destroyonlyswitch_core_file_closes the handles (which doesn't touchfh->event), so no double-free and no leak.Paired PR (must land together)
team-telnyx/mod_telnyxPR TEL-4617: Handle telnyx_force_cdr in respond event #69 consumesRecord-Open-Timeoutto emitfreeswitch_record_session_error_total{reason="timeout|other|unknown"}(and a separatefreeswitch_record_completion_total{cause}fromRECORD_STOPfor the mid-recording write/timeout mode). It degrades toreason="unknown"on cores without this change, so mod_telnyx can merge/deploy independently; this PR just upgradesunknown→timeout/other.Testing / limitations
telnyx_b2bua_builderpipeline.git diff --checkclean; confirmedhandle->eventhas no existing reader on this path; confirmedAVERROR(EINVAL/ENOMEM)are already used inavformat.csoAVERROR(ETIMEDOUT)resolves via the same libavutil includes; C90 decl placement checked.rw_timeoutset, e.g.uuid_record <uuid> start {rw_timeout=5000000}rtmp://unreachable:1935/x, and confirm therecord_session_errorevent now carriesRecord-Open-Timeout: true(fs_clievent trace or the mod_telnyx counter).