Skip to content

Template deletion does not remove build artifacts from storage (GCS/S3) #3239

Description

@AdaAibaby

Summary

When a user deletes a template via DELETE /templates/:templateId, the API only soft-deletes the database record. The associated build artifacts in GCS/S3 are never removed.

Steps to Reproduce

  1. Build a template and verify its build artifacts exist in storage.
  2. Delete the template via DELETE /templates/:templateId.
  3. Observe that the build objects remain in GCS/S3 indefinitely.

Expected Behavior

Template deletion should clean up build artifacts that are no longer referenced by any other template.

Actual Behavior

softDeleteTemplate() in api/internal/handlers/template_delete.go only performs DB operations:

  • SoftDeleteTemplate
  • ReleaseTemplateAliases
  • DeleteActiveTemplateBuilds

No storage cleanup (DeleteObjectsWithPrefix / DeleteObjects) is triggered at any point in the user-facing delete flow.

Evidence of Planned but Incomplete Implementation

The query db/queries/builds/get_exclusive_builds_for_template_deletion.sql already exists with the comment:

"Returns builds that are ONLY assigned to this template (safe to delete)"

However, this query has no Go callers — the storage cleanup logic that was meant to consume it was never implemented.

The only code path that calls DeleteObjectsWithPrefix on template builds is the internal TemplateBuildDelete gRPC handler, which is invoked only by admin build-cancel and a deprecated build-start flow — not by user template deletion.

Impact

  1. Storage leak: Deleted template artifacts accumulate indefinitely in GCS/S3, incurring ongoing storage costs.
  2. Runtime I/O errors (potential): If an external GCS lifecycle policy or manual GC later removes these objects, any sandboxes still using that build will encounter object does not exist errors propagating as NBD read failures and Firecracker disk I/O errors (EIO), crashing the sandbox filesystem.

Suggested Fix

On DELETE /templates/:templateId, after marking the template as deleted:

  1. Call GetExclusiveBuildsForTemplateDeletion to find build IDs solely owned by this template.
  2. For each such build, invoke TemplateBuildDelete (or call DeleteObjectsWithPrefix directly) to remove the GCS/S3 artifacts.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions