Skip to content

shim: bound the io drain in delete and release stdio when a created container stops - #14428

Open
Ayush-Rathor wants to merge 1 commit into
google:masterfrom
Ayush-Rathor:fix-shim-delete-unbounded-io-drain
Open

shim: bound the io drain in delete and release stdio when a created container stops#14428
Ayush-Rathor wants to merge 1 commit into
google:masterfrom
Ayush-Rathor:fix-shim-delete-unbounded-io-drain

Conversation

@Ayush-Rathor

Copy link
Copy Markdown

Fixes #14427.

Init.delete() drains the stdio copy goroutines and closes the pipes about twenty lines later:

p.wg.Wait()
...
if p.io != nil {
        for _, c := range p.closers { c.Close() }
        p.io.Close()
}

p.wg is released by copyPipes on EOF, which requires every write end to be closed. Init.Create
only passes the IO to the runtime when p.Sandbox, so for a sub-container the shim holds the write
ends itself and drops them only in CloseAfterStart(), after runsc start is successfully
launched. A sub-container that never started therefore blocks in p.wg.Wait() waiting for an EOF
that only the p.io.Close() below it can produce. That shim can never be reaped, and the next
containerd restart hangs in loadShims.

This is not specific to a particular failure. The precondition is only:

  1. runsc create succeeded — copyPipes runs after p.runtime.Create returns, so p.wg is +2
    (a failed create leaves it at zero and cannot deadlock);
  2. runsc start never succeeded, so CloseAfterStart() never ran;
  3. containerd eventually calls Delete.

1. Close the stdio on created -> stopped (init_state.go)

Leaving created means Start never succeeded, so the container never ran and has no output to
lose. Done in createdState.transition() rather than at call sites, so it covers every route out
of created — including createdState.State(), where a routine status query moves the state
machine to stopped without closing anything.

Deliberately not done when leaving running: such a container may still have buffered output in
flight, and closing early would truncate its logs, which is why delete() drains first and closes
after. This makes the s.p.io.Close() in createdState.Start()'s failure branch redundant, so it
is removed and the io is closed in exactly one place.

2. Bound the drain (init.go, exec.go, utils.go)

waitTimeout is restored from containerd's runc shim, which this package is forked from and which guards both drain sites with
waitTimeout(ctx, &p.wg, 10*time.Second).

This is needed on top of change 1 because createdState.Delete() calls p.delete() directly with
no transition at all, so a container created and deleted without Start ever being called still
reaches the unbounded wait. The timeout is self-healing rather than merely giving up: once
delete() stops waiting it proceeds to p.io.Close(), which releases the goroutines that were
blocking the drain. execProcess.delete() gains a ctx parameter to match; both callers in
exec_state.go already had one.

Testing

New pkg/shim/v1/proc/delete_test.go (5 tests). TestCreatedStateToStoppedClosesIO fails without
change 1:

io was not closed on the created->stopped transition; the shim would keep the
pipe write ends and Init.delete could never drain

//pkg/shim/... 23/23 pass. //runsc/container:container_test (8 shards) passes.

Reproduction steps and the affected-node evidence are in #14427.

What produced our particular route into this state is a separate sentry-side problem — after a
failed restore, the sandbox-wide l.state == restoreFailed is used to answer per-container status
queries, so a container created after the failure is reported RuntimeStateStopped on arrival
instead of RuntimeStateCreating. That needs its own fix and I will send it separately; this PR is
about making the shim survive.

…ontainer stops

Init.delete() waits for the stdio copy goroutines to drain and only closes
the pipes afterwards:

	p.wg.Wait()
	...
	if p.io != nil {
		for _, c := range p.closers { c.Close() }
		p.io.Close()
	}

p.wg is released by the two copyPipes goroutines, which return on EOF, and
EOF requires every write end to be closed. Init.Create only hands the IO to
the runtime for the sandbox:

	if p.Sandbox {
		opts.IO = p.io
	}

so for a sub-container the shim holds the write ends itself and releases
them only in CloseAfterStart(), which runs after runsc start has been
successfully launched. A sub-container that is created but never started
therefore leaves p.wg.Wait() waiting for an EOF that only the p.io.Close()
below it can produce. Such a shim can never be reaped, and a later
containerd restart blocks in loadShims and never finishes starting.

Note this is not specific to any one failure: the precondition is only that
runsc create succeeded (so copyPipes ran) and runsc start never did (so the
write ends were never handed over).

Two changes.

Close the stdio when a container leaves "created" for "stopped". Leaving
"created" means Start never succeeded, so the container never ran and has no
output to lose. This is done in createdState.transition() rather than at
individual call sites so that it covers every route out of "created" --
including createdState.State(), where a routine status query moves the state
machine to "stopped" without closing anything. It is deliberately not done
when leaving "running": a container that ran may still have buffered output
in flight, and closing early would truncate its logs, which is why delete()
drains first and closes after. This makes the existing s.p.io.Close() in
createdState.Start()'s failure branch redundant, so it is removed and the io
is now closed in exactly one place.

Bound the drain with waitTimeout. pkg/shim/v1/proc is derived from
containerd's runc shim (these files carry "Copyright 2018 The containerd
Authors"), which guards both drain sites with
waitTimeout(ctx, &p.wg, 10*time.Second); the bound was dropped when the code
was forked. It is needed on top of the first change because
createdState.Delete() calls p.delete() directly, with no transition at all,
so a container that is created and deleted without Start ever being called
still reaches the unbounded wait. The timeout is self-healing rather than
merely giving up: once delete() stops waiting it proceeds to p.io.Close(),
which releases the goroutines that were blocking the drain.
execProcess.delete() gains a ctx parameter to match; both callers in
exec_state.go already had one available.

Adds pkg/shim/v1/proc/delete_test.go. TestCreatedStateToStoppedClosesIO
fails without the first change with:

	io was not closed on the created->stopped transition; the shim would
	keep the pipe write ends and Init.delete could never drain
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Leftover containerd-shim-runsc-v1 processes after a failed restore make containerd hang forever on startup

1 participant