[Bug]: container cp on nonexistent source path hangs indefinitely; exec/stop on same container then hang too

[wooii]

I have done the following

• I have searched the existing issues
• If possible, I've reproduced the issue using the 'main' branch of this project

Steps to reproduce

1. container run -it --name test ubuntu:latest bash
2. container cp test:/existing/file.txt ~/local/file.txt
   → works fine, returns fast
3. container cp test:/path/that/does/not/exist ~/local/foo.txt
   → hangs indefinitely, no error, no timeout
4. Ctrl-C to abort
5. container stop test
   → also hangs indefinitely
6. container system stop
   → also hangs indefinitely

Only recovery found: restart macos.

Problem description

container cp on a source path that doesn't exist inside the container hangs
forever instead of returning an error immediately. Worse, once this happens every subsequent
command against that same container — exec -it, stop, system stop — also
hangs indefinitely.

Expected behavior:

• cp on a missing path should fail fast with a clear "no such file or
  directory" error.
• Even if a single operation does hang, it should not take down the whole
  container's control channel — stop should still work, or at minimum
  should time out with a clear error rather than hanging forever.

Actual behavior:

• cp hangs with no timeout.
• exec and stop on the same container then hang too.
• Only way to recover is restart macos.

Environment

- OS: macOS 26.5.2 (Tahoe)
- Hardware: Apple Silicon (M4)
- Container: Container CLI version 1.1.0 (installed via Homebrew)
- Image: docker.io/library/ubuntu:latest (Ubuntu 26.04 LTS, arm64)

Code of Conduct

• I agree to follow this project's Code of Conduct

[lakshitsoni26]

I dug into this and found the root cause! The hang actually isn't in this repo, it's upstream in apple/containerization.

When container cp gets a bad path, LinuxContainer.copyOut in the dependency holds an exclusive state lock and spins up a withThrowingTaskGroup to start the transfer and await an AsyncStream for the metadata. The bad path causes the first task to throw immediately, which cancels the group. But because AsyncStream iterators ignore task cancellation by default in Swift, the second task just sits there waiting forever for metadata that will never arrive.

Since the task group hangs, the LinuxContainer lock is held indefinitely, which is exactly why any subsequent exec or stop commands on that same container hang forever waiting for the lock.

I can pull the apple/containerization repo and open a PR over there to patch the AsyncStream continuations to properly call .finish() on cancellation if you'd like!

[wooii]

Hi @lakshitsoni26 , Thanks for the detailed root cause analysis. I can confirm that PR #1926 didn't fix the cp hang — I tested it and it still hangs the same way. If you open a PR in apple/containerization, I'd be happy to test it.

[stephenlclarke]

The lower-runtime fix is now proposed in apple/containerization#799. It terminates the copy metadata and vsock streams when guest validation fails, preserves the original guest-path error, and includes a VM regression that runs a successful exec on the same container afterward. Thanks @lakshitsoni26 for identifying the task-group/AsyncStream deadlock. @wooii, your offer to test would be especially useful on this PR.

[wooii]

Tested apple/containerization#799, I think #1927 is fixed. Details are here apple/containerization#799 (comment).
Thanks @stephenlclarke!

[lakshitsoni26]

Awesome! Glad my analysis helped point us in the right direction. Thanks for putting the fix together @stephenlclarke, and thanks @wooii for testing it!

[JeronimoColon]

I hit this independently while building on ContainerAPIClient and can confirm the whole chain on a stock 1.1.0 install: a copy from a guest path that does not exist hangs indefinitely, and stats, stop, kill, and delete --force on that container then hang behind it. My capture matches the root cause traced above - the stuck copy holds the container's state lock, so everything queued after it waits forever.

One recovery that works without restarting macOS: kill the wedged container's helper process. Each container gets its own, named for it:

ps -eo pid,command | grep container-runtime-linux
kill -9 <pid of the entry whose --uuid matches your container>

Every stuck call returns immediately (the pending cp with a timeout error, stop and delete with a connection error), the container lands cleanly in the stopped state, and it starts again afterward with its data intact. In my testing nothing else was affected - the daemon and other containers keep running.

Two halves worth keeping separate: apple/containerization#799 fixes the stall itself (tested working, per this thread), and #1862 would make the client's 300 second timeout actually fire, so even a stalled copy would fail with a clear error instead of hanging - today that timeout is selected internally but never propagates, which is why step 3 of the reproduction hangs with no error at all.