Skip to content

[fal.ai] longlive: LoRA file not found on session reinit — Civitai download races pipeline __init__ for [flux.2.klein] LoRA #937

@livepeer-tessa

Description

@livepeer-tessa

Summary

The longlive pipeline fails to load with "LoRA file not found" for [flux.2.klein]pixelart_redmond-000032.safetensors on the first pipeline load attempt. The error resolves itself on the next attempt once the LoRA download from Civitai completes. This is a race condition between asynchronous LoRA asset download and synchronous pipeline initialisation.

This is a distinct issue from:

cc @mjh1 @emranemran

Error Messages

2026-04-13 23:38:53,747 - scope.core.pipelines.wan2_1.lora.strategies.peft_lora - ERROR - LongLivePipeline.__init__: LoRA file not found: /tmp/.daydream-scope/assets/lora/[flux.2.klein]pixelart_redmond-000032.safetensors
2026-04-13 23:38:53,747 - scope.server.pipeline_manager - ERROR - Failed to load pipeline longlive: LongLivePipeline.__init__: LoRA loading failed. File not found: /tmp/.daydream-scope/assets/lora/[flux.2.klein]pixelart_redmond-000032.safetensors. If this error persists, consider removing the models directory '/data/models' and re-downloading models.
2026-04-13 23:38:53,957 - scope.server.pipeline_manager - ERROR - Some pipelines failed to load

Timeline (2026-04-13 ~23:35–23:40 UTC)

23:35:19 - httpx: HTTP Request: GET https://b2.civitai.com/.../5bflux2Klein5dpixelart.u4AF.safetensors
           (Civitai LoRA download begins)
23:36:06 - ✅ Pipeline yolo_mask loaded successfully (first job — LoRA already downloaded)
           Initial load params: loras=[{path: /tmp/.daydream-scope/assets/lora/[flux.2.klein]pixelart_redmond-000032.safetensors}]
23:36:34 - ✅ LongLivePipeline: Loaded LoRA '[flux.2.klein]pixelart_redmond-000032.safetensors' (first attempt, succeeds)

23:38:36 - Session resets (yolo_mask + longlive unloaded)
23:38:36 - pipeline_manager: Loading 2 pipeline(s): [yolo_mask, longlive]
23:38:53 - httpx: HTTP Request: GET https://b2.civitai.com/.../5bflux2Klein5dpixelart.u4AF.safetensors
           (Civitai download starts AFTER pipeline_manager already tries to load longlive)
23:38:53 - ❌ LoRA file not found: /tmp/.daydream-scope/assets/lora/[flux.2.klein]pixelart_redmond-000032.safetensors
           (file not yet downloaded from Civitai)
23:38:54 - scope_yolo_mask plugin installed
23:39:42 - ✅ LongLivePipeline: Loaded LoRA '[flux.2.klein]pixelart_redmond-000032.safetensors' (retry, succeeds)

Job IDs:

  • ea7914a1-6a2f-4472-ad7b-e6fc0983cde2 (no app label, grpc-run)
  • a8a03ca5-6fce-4cdd-8bca-580e8fbafeeb (scope-app--prod)

LoRA file: [flux.2.klein]pixelart_redmond-000032.safetensors (source: Civitai)
Frequency (last 12h): 1 session (23:36–23:40 UTC 2026-04-13)

Root Cause

When a session is set up with a Civitai-hosted LoRA, the file download is initiated but the pipeline's __init__ proceeds synchronously to load the LoRA before the download has completed. This causes a spurious FileNotFoundError on session reinitialisation (the second job).

On the first job, the LoRA download had already completed from a prior session, so it was available. On the second job restart, the tmp/ state was partially cleared or the download re-triggered, causing the race.

Impact

Suggested Fix

Ensure LoRA assets are fully downloaded to /tmp/.daydream-scope/assets/lora/ before LongLivePipeline.__init__ attempts to load them. Options:

  1. Await all asset downloads before invoking pipeline __init__
  2. Add a polling retry in peft_lora for the file to appear (with timeout)
  3. Pre-seed the download on session setup so it's ready when the pipeline loads

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions