Skip to content

api: "Node exhausted" placement warning drops the exhaustion reason from logs #3278

Description

@AdaAibaby

Describe the bug

When PlaceSandbox receives codes.ResourceExhausted from an orchestrator node it logs:

WARN  Node exhausted, trying another node  {"sandbox.id": "...", "node.id": "cpu-nat-398"}

The gRPC error — which contains the specific reason — is silently dropped. The orchestrator has two distinct ResourceExhausted triggers with different meanings:

Condition Error message
Running sandboxes ≥ MaxSandboxesPerNode "max number of running sandboxes on node reached (N), please retry"
Concurrent starts semaphore full (MaxStartingInstancesPerNode, default 3) "too many sandboxes starting on this node, please retry"

Without the error in the log it is impossible to tell from the API side which limit was hit. This makes diagnosing capacity vs. throughput issues significantly harder.

Root cause

placement.go line 152:

case codes.ResourceExhausted:
    failedNode.PlacementMetrics.Skip(sbxRequest.GetSandbox().GetSandboxId())
    logger.L().Warn(ctx, "Node exhausted, trying another node",
        logger.WithSandboxID(sbxRequest.GetSandbox().GetSandboxId()),
        logger.WithNodeID(failedNode.ID),
        // ← err is never logged here
    )

The default branch immediately below already uses zap.Error(utils.UnwrapGRPCError(err)), but it was omitted from the ResourceExhausted case.

Expected behavior

WARN  Node exhausted, trying another node  {
  "sandbox.id": "i5kfsfuncpgnkn65a5bue",
  "node.id": "cpu-nat-398",
  "error": "too many sandboxes starting on this node, please retry"
}

Suggested fix

Add zap.Error(utils.UnwrapGRPCError(err)) to the ResourceExhausted Warn call, matching the existing pattern in the default branch.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions