[Bug]: pre-update lifecycle hook fails — ExecStart blocks then ExecAttach captures "is already running" (exit 126) on 1.17.x / Docker v29

[claesjonsson]

Bug Report Checklist

• I checked existing issues and found no duplicates.
• I have reviewed Watchtower's documentation and confirmed compliance with documented configuration specifications.
• I have restarted Docker, re-pulled the latest image, and confirmed that the issue persists.

Bug Description

Summary: Since 1.17.x (the docker/docker → moby/moby v29 migration, #1613), every container with a pre-update lifecycle hook fails its update, even though the hook command itself runs successfully.

Root cause (from debug log): In ExecuteCommand, watchtower calls ExecStart(exec.ID) and then captureExecOutput → ExecAttach(exec.ID) on the same exec instance. Under the v29 client, ExecStart (Detach=false) now runs the command synchronously to completion — the log shows a 24s gap between "Starting exec instance" and "Attaching to exec instance," matching the command's runtime. The subsequent ExecAttach therefore hits an already-finished exec; the daemon returns exec command is already running, which watchtower captures as the command's output, and the exec is then reported as exit_code=126. The hook is marked failed and the update aborted, despite the actual command succeeding.

Expected: capture the real stdout/exit code of the pre-update command and proceed with the update.

Suggested fix: don't start-then-attach. Either attach via the hijacked stream to capture output as the command runs, or start detached (Detach: true) and rely solely on ExecInspect for the exit code.

Versions: watchtower 1.17.2, Docker Engine v29.x.
Regression boundary: fails on 1.17.x; passes on 1.16.0 (same engine, same container).

See debug log below

Repro: any container with com.centurylinklabs.watchtower.lifecycle.pre-update whose hook runs long enough to make the timing obvious (even a sleep 5; echo done works). See repro steps below.

Steps to Reproduce

Steps to reproduce

Host: Docker Engine 29.x, watchtower 1.17.2.

1. Make watchtower believe an update is pending for a test container by
   retagging an older image as latest:

   docker pull nginx:1.27.0
   docker tag nginx:1.27.0 nginx:latest

2. Start a target container that has a pre-update lifecycle hook. The hook
   does nothing but echo — its content is irrelevant to the bug:

   docker run -d --name repro-target
   --label com.centurylinklabs.watchtower.enable=true
   --label com.centurylinklabs.watchtower.lifecycle.pre-update="sleep 10; echo pre-update-hook-ran"
   nginx:latest

3. Run watchtower once with lifecycle hooks enabled, scoped to that container:

   docker run --rm
   -e WATCHTOWER_LIFECYCLE_HOOKS=true
   -v /var/run/docker.sock:/var/run/docker.sock
   nickfedor/watchtower:1.17.2 --run-once repro-target

Expected

Watchtower runs the pre-update hook, then proceeds to update repro-target
to the newer nginx:latest.

Actual

The hook command runs (you can see pre-update-hook-ran), but watchtower then
fails and aborts the update:

Updated container state to failed | container_id=
error=pre-update command failed: pre-update command execution failed for
container repro-target: failed to inspect exec instance: command execution
failed with exit code 126: exec command has already run

Expected Behavior

Watchtower runs the pre-update hook, then proceeds to update repro-target
to the newer nginx:latest

Watchtower Version

1.17.2

Docker Version

29.4.1

Docker Info (optional)

Client: Docker Engine - Community
 Version:    29.4.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.33.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.1.3
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  model: Docker Model Runner (Docker Inc.)
    Version:  v1.1.37
    Path:     /usr/libexec/docker/cli-plugins/docker-model

Server:
 Containers: 11
  Running: 11
  Paused: 0
  Stopped: 0
 Images: 289
 Server Version: 29.4.1
 Storage Driver: zfs
  Zpool: nmt
  Zpool Health: ONLINE
  Parent Dataset: nmt/docker
  Space Used By Parent: 99044120448
  Space Available: 17900472622304
  Parent Quota: no
  Compression: lz4
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 77c84241c7cbdd9b4eca2591793e3d4f4317c590
 runc version: v1.3.5-0-g488fc13e
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.0-107-generic
 Operating System: Ubuntu 24.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 62.67GiB
 Name: sinclair
 ID: REDACTED
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Firewall Backend: iptables

Docker Compose Configuration

services:
  watchtower:
    image: nickfedor/watchtower:latest
    container_name: watchtower
    restart: unless-stopped
    environment:
      TZ: "Europe/Stockholm"
      WATCHTOWER_SCHEDULE: "0 0 4 * * *"
      WATCHTOWER_LABEL_ENABLE: "true"
      WATCHTOWER_CLEANUP: "true"
      WATCHTOWER_NOTIFICATION_URL: /run/secrets/slack_url
      WATCHTOWER_LIFECYCLE_HOOKS: "true"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    secrets:
      - slack_url
    labels:
      com.centurylinklabs.watchtower.enable: "true"

secrets:
  slack_url:
    file: ./.slack_url

Docker CLI Command

Debug Logs

time="...17:44:53..." level=debug msg="Executing pre-update command" command=/before_update.sh container=gitlab timeout=5
time="...17:44:53..." level=debug msg="Creating exec instance" command=/before_update.sh container_id=069936a77cd9d70b...
time="...17:44:53..." level=debug msg="Starting exec instance" container_id=069936a77cd9... exec_id=c7072f5d48be...
time="...17:45:17..." level=debug msg="Attaching to exec instance" exec_id=c7072f5d48be...
time="...17:45:17..." level=debug msg="Captured exec output" exec_id=c7072f5d48be... output="exec command c7072f5d48be... is already running"
time="...17:45:17..." level=debug msg="Checked exec status" exec_id=c7072f5d48be... exit_code=0 running=true
time="...17:45:18..." level=debug msg="Checked exec status" exec_id=c7072f5d48be... exit_code=126 running=false
time="...17:45:18..." level=debug msg="Command output captured" exec_id=c7072f5d48be... output_length=96
time="...17:45:18..." level=debug msg="Command execution failed" error="command execution failed with exit code 126: exec command c7072f5d48be... is already running" exec_id=c7072f5d48be...
time="...17:45:18..." level=debug msg="Failed to inspect exec instance" container_id=069936a77cd9... error="command execution failed with exit code 126: exec command c7072f5d48be... is already running"
time="...17:45:18..." level=warning msg="Updated container state to failed" container_id=069936a77cd9 error="pre-update command failed: ... exec command c7072f5d48be... is already running" name=gitlab

Additional Context

Note the 24 second gap:

17:44:53 Starting exec instance exec_id=c7072f...
17:45:17 Attaching to exec instance exec_id=c7072f... ← 24 seconds later - the pre-update command happens to take roughly 24 seconds

ExecStart isn't just starting the command, it's running it synchronously to completion. By the time watchtower calls ExecAttach to "capture output," the command is already finished, and the daemon responds to the attach with the conflict string:
output="exec command c7072f... is already running"

After the moby/moby v29 migration (#1613, 1.17.x), ExecStart now blocks/runs the command itself, so the attach-afterward pattern always lands on a finished exec.