Skip to content

TELCORE-355: propagate concrete recorder open-error detail on record_session_error - #638

Draft
minhtuan1407-telnyx wants to merge 2 commits into
telnyx/telephony/deploy-developmentfrom
tuan/telcore-355-record-open-error-detail
Draft

TELCORE-355: propagate concrete recorder open-error detail on record_session_error#638
minhtuan1407-telnyx wants to merge 2 commits into
telnyx/telephony/deploy-developmentfrom
tuan/telcore-355-record-open-error-detail

Conversation

@minhtuan1407-telnyx

Copy link
Copy Markdown

What

Propagate the concrete recorder open-error (and an explicit timeout flag) on the record_session_error event, so downstream metrics/alerting can distinguish a recorder-server timeout from any other recording open failure.

Linear: https://linear.app/telnyx/issue/TELCORE-355/add-b2bua-alerting-for-recorder-server-timeouts

Why

In the 2026-08-07 incident (~170k calls affected), recorder-server RTMP timeouts manifested as:

avformat.c:2331  Could not open 'rtmp://recorder.query.prod.telnyx.io:1935/...': Connection timed out
switch_ivr_async.c:3735  Error opening {stereo=false,rw_timeout=15000000,...}rtmp://recorder...

The record_session_error CUSTOM event fired, but it only carried Record-Error = "Error opening file" — the specific cause ("Connection timed out" vs connection-refused / DNS / other) lived only in the mod_av/avformat.c log line, never in the event. So mod_telnyx (the Prometheus scrape endpoint) could not build a timeout-specific counter/alert without a core change.

Change

  • mod_av (avformat.c): on avio_open2/avio_open failure, stash the concrete error text and an explicit timeout flag on the otherwise-unused handle->event:
    • Record-Open-Error-Detail = av_strerror text
    • Record-Open-Timeout = true iff ret == AVERROR(ETIMEDOUT) (exact code check — not string matching; ETIMEDOUT is precisely how the rw_timeout interrupt callback surfaces).
  • switch_ivr_async.c: send_record_error_event() gains a switch_file_handle_t *fh param and forwards those as Record-Error-Detail / Record-Open-Timeout headers when present. The three record_session open-failure call sites pass the handle and switch_event_destroy(&fh->event) after use.

Headers are absent for file modules that don't populate them (WAV, sndfile, etc.) → fully backwards compatible; existing consumers see the same Record-Error they always did.

Lifetime / safety notes

  • handle->event is unused on the recording open path today (git grep confirms no reader in switch_ivr_async.c record_session). It is a SWITCH_EVENT_CLONE event with its own allocation, not tied to fh->memory_pool — so it survives switch_core_file_open's fail: cleanup (which only destroys fh->params and, when it owns it, fh->memory_pool).
  • Each open-failure call site destroys the event right after firing; record_helper_destroy only switch_core_file_closes the handles (which doesn't touch fh->event), so no double-free and no leak.
  • C90: new declarations are at the top of their block scope.

Paired PR (must land together)

  • team-telnyx/mod_telnyx PR TEL-4617: Handle telnyx_force_cdr in respond event #69 consumes Record-Open-Timeout to emit freeswitch_record_session_error_total{reason="timeout|other|unknown"} (and a separate freeswitch_record_completion_total{cause} from RECORD_STOP for the mid-recording write/timeout mode). It degrades to reason="unknown" on cores without this change, so mod_telnyx can merge/deploy independently; this PR just upgrades unknowntimeout/other.

Testing / limitations

  • No local compile in this environment (FreeSWITCH core requires the full autotools build tree). The compile gate is the standard FreeSWITCH / telnyx_b2bua_builder pipeline.
  • Verification performed here: git diff --check clean; confirmed handle->event has no existing reader on this path; confirmed AVERROR(EINVAL/ENOMEM) are already used in avformat.c so AVERROR(ETIMEDOUT) resolves via the same libavutil includes; C90 decl placement checked.
  • Manual staging: point a recording at an unreachable RTMP host with rw_timeout set, e.g. uuid_record <uuid> start {rw_timeout=5000000}rtmp://unreachable:1935/x, and confirm the record_session_error event now carries Record-Open-Timeout: true (fs_cli event trace or the mod_telnyx counter).

minhtuan1407-telnyx added 2 commits August 11, 2026 16:18
…session_error

When a record_session open fails (e.g. the recorder-server RTMP timeout in the
2026-08-07 incident), the record_session_error CUSTOM event only carried
Record-Error="Error opening file", with the specific cause (Connection timed
out vs connection refused/DNS/other) visible only in the mod_av log line. This
made it impossible for downstream metrics/alerting (mod_telnyx) to distinguish
a recorder-server timeout from any other open failure.

- mod_av (avformat.c): on avio_open2/avio_open failure, stash the concrete
  error text and an explicit rw_timeout flag on the otherwise-unused
  handle->event (Record-Open-Error-Detail, Record-Open-Timeout). The
  ETIMEDOUT check is exact (AVERROR(ETIMEDOUT)), not string matching.
- switch_ivr_async.c: send_record_error_event() forwards those as
  Record-Error-Detail / Record-Open-Timeout headers on the event when the file
  handle carries them; the three record_session open-failure call sites pass
  the file handle and destroy handle->event after use. Headers are absent for
  file modules that don't populate them (backwards compatible).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant