Skip to content

Remote BuildKit volume grows unbounded, no built-in pruning #1794

@yshmarov

Description

@yshmarov

Problem

When using builder.remote, Kamal creates a BuildKit container on the target server and reuses it across deploys for layer caching. However, the BuildKit volume grows with every deploy and is never pruned by Kamal.

On a small VPS (e.g., Hetzner CPX22 — 80 GB disk), the BuildKit state volume can grow to 40–50 GB over a few weeks of regular deploys, consuming most of the available disk space.

Critically, the existing kamal prune command and docker system prune do not reclaim this space — the data lives inside a named Docker volume (buildx_buildkit_*_state), not in the build cache layer that docker buildx prune targets.

Current workaround

A weekly cron job to cap the cache:

0 5 * * 0 docker buildx prune --keep-storage=5GB -f >> /var/log/docker-cleanup.log 2>&1

If the volume has already grown too large, the only fix is to remove the BuildKit container and volume entirely (losing all cache):

docker stop $(docker ps -qf name=buildkit) && docker rm $(docker ps -aqf name=buildkit) && docker volume rm $(docker volume ls -q --filter name=buildkit)

Suggestion

A few options that could address this at the Kamal level:

  1. Add a builder.cache.max_storage option in deploy.yml — Kamal would run docker buildx prune --keep-storage=<value> -f after each deploy or as part of kamal prune
  2. Include BuildKit pruning in kamal prune — the existing prune command handles old images and containers but not BuildKit
  3. Document the issue — at minimum, mention that remote BuildKit volumes grow unbounded and suggest a cron-based workaround

Environment

  • Kamal 2
  • builder.remote: ssh://root@<ip> (remote build on same VPS)
  • Hetzner CPX22 (2 vCPU, 4 GB RAM, 80 GB disk)
  • Observed BuildKit volume size after ~2 weeks of deploys: 48 GB

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions