[fal.ai] Remote record proof validation fails: source.mp4 not found at /tmp/daydream-remote-record-proof/

[livepeer-tessa]

Summary

The remote GPU path record validation check fails because the expected file /tmp/daydream-remote-record-proof/source.mp4 does not exist on the fal.ai worker. The job appears to be an automated passthrough test, but the test file is never placed at the expected path, causing the video_file input to error immediately.

cc @mjh1 @emranemran

Error Logs (Grafana/Loki — 2026-04-11 ~12:14 UTC)

Session ID: 0cd561ee
fal_app_name: github_f1lhgmk5v76a0ev1w0u378by-scope-app--prod

2026-04-11 12:14:04,609 - scope.core.inputs.video_file - ERROR - [0cd561ee] Video file not found: /tmp/daydream-remote-record-proof/source.mp4
2026-04-11 12:14:04,609 - scope.server.source_manager - ERROR - [0cd561ee] Failed to connect input source video_file for node input

Job Parameters (from logs just before)

{
  "input_mode": "video",
  "prompts": [{"text": "remote GPU path record validation passthrough", "weight": 1.0}],
  "pipeline_ids": ["passthrough"],
  "graph": {
    "nodes": [
      {
        "id": "input",
        "type": "source",
        "source_mode": "video_file",
        "source_name": "/tmp/daydream-remote-record-proof/source.mp4",
        "tempo_sync": false
      }
    ]
  }
}

Analysis

This is an automated remote record proof validation job. The job expects a test video to already exist at /tmp/daydream-remote-record-proof/source.mp4 on the worker, but fal.ai workers are ephemeral — the file is not present.

Possible root causes:

1. The validation setup step that creates the test file is not running (or not running before the Scope job starts)
2. The file path is hardcoded but the directory is created by a separate process that may have a race condition or fail silently
3. The /tmp/daydream-remote-record-proof/ directory itself may not exist

Impact

• Automated record proof validation fails silently with ERROR-level logs
• If this is a health check, the check is not actually validating GPU path recording capability
• scope.server.source_manager cannot connect the input, so the session starts in a degraded state

Suggested Fix

Ensure the validation setup creates the directory and places a test video file before the Scope job session starts, or have the video_file input handle the missing file case with a clearer error and early exit that surfaces the failure explicitly.