Describe the bug
When PlaceSandbox receives codes.ResourceExhausted from an orchestrator node it logs:
WARN Node exhausted, trying another node {"sandbox.id": "...", "node.id": "cpu-nat-398"}
The gRPC error — which contains the specific reason — is silently dropped. The orchestrator has two distinct ResourceExhausted triggers with different meanings:
| Condition |
Error message |
Running sandboxes ≥ MaxSandboxesPerNode |
"max number of running sandboxes on node reached (N), please retry" |
Concurrent starts semaphore full (MaxStartingInstancesPerNode, default 3) |
"too many sandboxes starting on this node, please retry" |
Without the error in the log it is impossible to tell from the API side which limit was hit. This makes diagnosing capacity vs. throughput issues significantly harder.
Root cause
placement.go line 152:
case codes.ResourceExhausted:
failedNode.PlacementMetrics.Skip(sbxRequest.GetSandbox().GetSandboxId())
logger.L().Warn(ctx, "Node exhausted, trying another node",
logger.WithSandboxID(sbxRequest.GetSandbox().GetSandboxId()),
logger.WithNodeID(failedNode.ID),
// ← err is never logged here
)
The default branch immediately below already uses zap.Error(utils.UnwrapGRPCError(err)), but it was omitted from the ResourceExhausted case.
Expected behavior
WARN Node exhausted, trying another node {
"sandbox.id": "i5kfsfuncpgnkn65a5bue",
"node.id": "cpu-nat-398",
"error": "too many sandboxes starting on this node, please retry"
}
Suggested fix
Add zap.Error(utils.UnwrapGRPCError(err)) to the ResourceExhausted Warn call, matching the existing pattern in the default branch.
Describe the bug
When
PlaceSandboxreceivescodes.ResourceExhaustedfrom an orchestrator node it logs:The gRPC error — which contains the specific reason — is silently dropped. The orchestrator has two distinct
ResourceExhaustedtriggers with different meanings:MaxSandboxesPerNode"max number of running sandboxes on node reached (N), please retry"MaxStartingInstancesPerNode, default 3)"too many sandboxes starting on this node, please retry"Without the error in the log it is impossible to tell from the API side which limit was hit. This makes diagnosing capacity vs. throughput issues significantly harder.
Root cause
placement.goline 152:The
defaultbranch immediately below already useszap.Error(utils.UnwrapGRPCError(err)), but it was omitted from theResourceExhaustedcase.Expected behavior
Suggested fix
Add
zap.Error(utils.UnwrapGRPCError(err))to theResourceExhaustedWarn call, matching the existing pattern in thedefaultbranch.