Skip to content

[fal.ai/livepeer-staging] Trickle subscriber GET fails mid-session — sub cannot complete GET on segment /10 #915

@livepeer-tessa

Description

@livepeer-tessa

Summary

The livepeer_gateway.trickle_subscriber is throwing an error during an active session while trying to GET a trickle segment from the orchestrator. This is distinct from the publisher errors (#912, #846) — the subscriber side (reading processed output back from the orch) is failing mid-session.

cc @mjh1 @emranemran

Error Logs (Grafana/Loki — 2026-04-11 ~14:06 UTC)

Session manifest_id: 2153e39d
fal_app_name: github_f1lhgmk5v76a0ev1w0u378by-scope-livepeer

2026-04-11 14:06:55,287 - livepeer_gateway.trickle_subscriber - ERROR - Trickle sub failed to complete GET https://orch-staging-1.daydream.monster:8935/ai/trickle/2153e39d-1-in/10

  File "/app/.venv/lib/python3.12/site-packages/livepeer_gateway/trickle_subscriber.py", line 150, in _preconnect

Context

  • Session started at 14:00:57 UTC (subscribing to control channel 2153e39d-control)
  • Error occurs ~6 minutes in at 14:06:55 UTC, on segment /10 of the 2153e39d-1-in stream
  • The session was actively publishing successfully before this point (seen from the MediaPublishStats logs on the companion scope-livepeer job running at the same time: 300+ segments completed, 0 failed)

Analysis

The subscriber is attempting to GET the orchestrator's input segment /10 but the GET fails. This could be:

  1. A network blip between the fal.ai worker and orch-staging-1.daydream.monster:8935
  2. The orchestrator dropped the trickle stream / stream not found on that endpoint
  3. A timeout in _preconnect before the segment is available

The failure occurs on the input stream (-1-in) rather than the output, suggesting the subscriber is feeding processed video back into the session pipeline and the orch-side endpoint has gone away.

Affected Components

  • livepeer_gateway.trickle_subscriber._preconnect (line 150)
  • Orchestrator: orch-staging-1.daydream.monster:8935

Impact

  • Subscriber cannot receive processed frames from orchestrator
  • Session may continue running but output will stall/degrade
  • No automatic recovery observed in logs

Related Issues

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions