Skip to content

Worker occasionally dies with "UpdateForWorker stream closed early" #2115

@rejuvenile

Description

@rejuvenile
  2026-01-21T20:58:46.749339Z ERROR nativelink_worker::running_actions_manager: RunningActionImpl did not cleanup. This is a violation of the requirements, will attempt to do it in the background., operation_id: 9c671a53-b7fc-4c2b-ba51-5639f1b064b5
    at nativelink-worker/src/running_actions_manager.rs:1415
    in nativelink_worker::local_worker::worker_start_action
    in nativelink_worker::local_worker::worker_start_action_ctx with operation_id: "9c671a53-b7fc-4c2b-ba51-5639f1b064b5", digest_function: BLAKE3
    in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
    in nativelink::worker with name: "worker_0"

  2026-01-21T20:58:46.749389Z ERROR nativelink_worker::running_actions_manager: RunningActionImpl did not cleanup. This is a violation of the requirements, will attempt to do it in the background., operation_id: 6cac1188-bff7-4038-996f-4612bb138fe2
    at nativelink-worker/src/running_actions_manager.rs:1415
    in nativelink_worker::local_worker::worker_start_action
    in nativelink_worker::local_worker::worker_start_action_ctx with operation_id: "6cac1188-bff7-4038-996f-4612bb138fe2", digest_function: BLAKE3
    in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
    in nativelink::worker with name: "worker_0"

  2026-01-21T20:58:59.487142Z ERROR nativelink_worker::local_worker: ERROR_MSG: "Actions in transit did not reach zero before we disconnected from the scheduler"
    at nativelink-worker/src/local_worker.rs:701
    in nativelink_worker::local_worker::run with shutdown_rx: broadcast::Receiver
    in nativelink::worker with name: "worker_0"


thread 'main' (24873374) panicked at src/bin/nativelink.rs:702:9:
Error { code: Internal, messages: ["UpdateForWorker stream closed early", "Actions in transit did not reach zero before we disconnected from the scheduler"] }

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions