Skip to content

TELCORE-355: report recorder open timeouts on record_session_error - #642

Open
minhtuan1407-telnyx wants to merge 1 commit into
telnyx/telephony/masterfrom
linear-telcore-355
Open

TELCORE-355: report recorder open timeouts on record_session_error#642
minhtuan1407-telnyx wants to merge 1 commit into
telnyx/telephony/masterfrom
linear-telcore-355

Conversation

@minhtuan1407-telnyx

@minhtuan1407-telnyx minhtuan1407-telnyx commented Aug 13, 2026

Copy link
Copy Markdown

Problem

switch_core_file_open() collapsed every open failure into SWITCH_STATUS_GENERR. When the recorder server timed out during INC-217 the only place the concrete cause appeared was the log line:

avformat.c:2331 Could not open 'rtmp://recorder.query.prod.telnyx.io:1935/mp3/...': Connection timed out
switch_ivr_async.c:3735 Error opening {...rw_timeout=15000000...}rtmp://recorder...

The record_session_error event that follows carries only Record-File-Path and a fixed Record-Error: Error opening file, so a recorder-server timeout is indistinguishable from a bad URL, a refused connection or a full disk. All B2BUA alerting in tel-core-alerts is Prometheus-expression based and cannot read logs, so the distinction has to reach the event to be alertable.

Change

Two files, +18/-8.

  • mod_av: av_file_open() returns SWITCH_STATUS_TIMEOUT instead of SWITCH_STATUS_GENERR when avio_open2() fails with AVERROR(ETIMEDOUT) — i.e. rw_timeout expiry.
  • switch_ivr_async: capture the open status at the three record-open failure sites (single file, split in, split out) and add Record-Open-Timeout: true|false to record_session_error.

Why the status and not a handle field or handle->event

The status is the only channel that already survives a failed open. switch_core_perform_file_open()'s fail: path destroys fh->params and the handle memory pool (switch_core_file.c), so nothing allocated there lives past the return; and it propagates the file module's status unchanged, so a distinct status arrives at the caller for free.

fh->event does survive, but it is caller-owned playback metadata merged into PLAYBACK_START/PLAYBACK_STOP (switch_ivr_play_say.c:1685, :2087) and read by mod_http_cache, so a format module writing into it means either polluting caller data or handing ownership out of a failed open. A new field on switch_file_handle_t would work but changes the layout of a struct that callers all over the tree stack-allocate — an avoidable rebuild hazard for a boolean that the return value can already carry.

Blast radius

  • send_record_error_event() is static with exactly three call sites, all updated.
  • All 92 switch_core_file_open() call sites in the tree were checked: every one branches on SWITCH_STATUS_SUCCESS (== or !=) or propagates the status to a caller that does, so a second failure status is inert for them. The one test that asserts a specific failure status, avformat_test_read_err (test_avformat.c:208, expects GENERR), is on the read path — this change is in the output-file branch of av_file_open() and cannot reach it.
  • Every success path is unchanged; the event gains one header.

Verification

  • Built with the paired mod_telnyx change: USE_REGISTRY_IMAGE=1 make build-deb-package-on-docker in telnyx_b2bua_builder (telnyx_b2bua-1.10.12-telv111.1.deb).
  • Runtime check (canary): point a recording at an unroutable recorder and confirm the header, then the counter it feeds:
fs_cli -x "uuid_record <uuid> start {rw_timeout=5000000,modname=mod_av}rtmp://10.255.255.1:1935/mp3/x"
# record_session_error now carries Record-Open-Timeout: true
curl -s localhost:7080/metrics | grep freeswitch_record_failure_total

An unreachable-but-routable target (connection refused) should report Record-Open-Timeout: false, which is the other half of the check.

Paired PR

Consumer: https://github.com/team-telnyx/mod_telnyx/pull/72 — counts the classified failures into freeswitch_record_failure_total. That PR is independently deployable; without this one every open failure lands under reason="open-unknown".

Linear: https://linear.app/telnyx/issue/TELCORE-355/add-b2bua-alerting-for-recorder-server-timeouts

switch_core_file_open() collapsed every open failure into SWITCH_STATUS_GENERR,
so a recorder-server timeout (rw_timeout expiry against rtmp://recorder...)
was indistinguishable from a bad URL, a refused connection or a full disk.
The concrete cause only ever reached the log line, which the
Prometheus-based alerting in tel-core-alerts cannot read.

mod_av now returns SWITCH_STATUS_TIMEOUT for an ETIMEDOUT open.
switch_core_perform_file_open() propagates the file module's status
unchanged through its failure path, so switch_ivr_async can report it as
Record-Open-Timeout: true|false on record_session_error and mod_telnyx
can count recorder timeouts separately (TELCORE-355).

The status is the only channel that already survives a failed open:
fh->params and the handle pool are destroyed on that path, and fh->event
is caller-owned playback metadata. Every caller of switch_core_file_open()
tests against SWITCH_STATUS_SUCCESS, so a second failure status is inert
for them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@dev-ryanc dev-ryanc left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PRBot automated review — no critical issues found. Approved based on: small mechanical change (18 additions, 8 deletions), error-path only — captures switch_core_file_open status and adds Record-Open-Timeout header to record_session_error event. Verified switch_core_perform_file_open passes module status through fail: path (line 396→469). No control flow change, no call-path impact on success path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants