Skip to content

feat: Track history in per layer metadata file #568

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rossdylan wants to merge 2 commits into
base: main
Choose a base branch
from
+121 −8
Open

feat: Track history in per layer metadata file #568

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5d84aa5
feat: Track history in per layer metadata file
rossdylan Jun 19, 2026
0d5445a
Fix, extend history propagation
rossdylan Jun 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions img/private/import.bzl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,13 +52,21 @@ def _write_layer_info(ctx, manifest, config, layer_index, index_position = None)
config.get("architecture", "unknown"),
layer_index,
)

# History is stored in the config, with one entry per layer
config_history = config.get("history")
if config_history and len(config_history) > layer_index:
layer_history = config_history[layer_index]
else:
layer_history = None

@malt3 malt3 Jun 19, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is not good enough.
The real rule for this is that we may have:

  • No history (you handle that)
  • too little history (you handle that)
  • Up to len(layers) layer with empty_layer == false
  • In between those, any number of history entries with "empty_layer": true

We need to align the non-empty history entries to the layers and use some heuristic to assign the empty_layer history entries. Either we assign the empty_layer entries to the previous layer or the next layer (doesn't really matter, pick one). This also means that the metadata for a single layer must hold a list of history entries, where only one member may be empty_layer false. Still, in most cases we will have exactly one entry. :)

@rossdylan rossdylan Jun 19, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah okay I think I finally understand, took me a second to remember that things like ENV won't actually generate an on-disk layer. I'll update to collapse the empty layer histories into the layer before it.

@rossdylan rossdylan Jun 19, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have any pointers to running the specific integration test that gets tripped up locally? Don't want to waste your time doing testing round trips.

@malt3 malt3 Jun 19, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bazel query e2e:all
bazel test (any name that comes back)

@malt3 malt3 Jun 19, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also:

cd e2e/go
bazel run --platforms=//platform:linux_amd64 image:load

metadata = dict(
name = name,
diff_id = diff_id,
mediaType = media_type,
digest = digest,
size = size,
annotations = layer.get("annotations", {}),
history = layer_history,
)
index_position_str = "" if index_position == None else str(index_position) + "_"
layer_metadata = ctx.actions.declare_file(ctx.attr.name + "_{}{}_layer_metadata.json".format(index_position_str, layer_index))
Expand Down
9 changes: 9 additions & 0 deletions img_tool/cmd/manifest/manifest.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -338,6 +338,15 @@ func prepareConfig(layers []api.Descriptor, templatesData *ConfigTemplates, crea
if createdTime != nil {
config.Created = createdTime
}
for _, layer := range layers {
config.History = append(config.History, specv1.History{
Created: layer.History.Created,
CreatedBy: layer.History.CreatedBy,
Author: layer.History.Author,
Comment: layer.History.Comment,
EmptyLayer: layer.History.EmptyLayer,
})
}

return config, nil
}
Expand Down
21 changes: 21 additions & 0 deletions img_tool/pkg/api/api.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import (
"io"
"io/fs"
"iter"
"time"
)

type (
Expand Down Expand Up @@ -65,13 +66,33 @@ var (
Symlink = FileType{"l"}
)

// History describes the history of a layer.
// This is a re-export of the oci spec v1 History structure to avoid taking a dep on it
type History struct {
// Created is the combined date and time at which the layer was created, formatted as defined by RFC 3339, section 5.6.
Created *time.Time `json:"created,omitempty"`

// CreatedBy is the command which created the layer.
CreatedBy string `json:"created_by,omitempty"`

// Author is the author of the build point.
Author string `json:"author,omitempty"`

// Comment is a custom message set when creating the layer.
Comment string `json:"comment,omitempty"`

// EmptyLayer is used to mark if the history item created a filesystem diff.
EmptyLayer bool `json:"empty_layer,omitempty"`
}

type Descriptor struct {
Name string `json:"name,omitempty"`
DiffID string `json:"diff_id,omitempty"`
MediaType string `json:"mediaType"`
Digest string `json:"digest"`
Size int64 `json:"size"`
Annotations map[string]string `json:"annotations,omitempty"`
History History `json:"history"`
}

type AppenderState struct {
Expand Down
1 change: 1 addition & 0 deletions img_tool/pkg/metadata/metadata.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ func WriteLayerMetadata(
Digest: digest,
Size: size,
Annotations: mergedAnnotations,
History: api.History{CreatedBy: name},
}

encoder := json.NewEncoder(outputFile)
Expand Down
3 changes: 2 additions & 1 deletion tests/img_toolchain/testcases/compress_with_annotations.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,4 +28,5 @@ file_contains = compress-meta.json, "team"
file_contains = compress-meta.json, "platform"
file_contains = compress-meta.json, "project"
file_contains = compress-meta.json, "rules_img"
file_sha256 = compress-meta.json, "7b492556ebc6c8791f42b36d67adc861fd3751a18ddde88cf150538435aab80c"
file_contains = compress-meta.json, "history"
file_sha256 = compress-meta.json, "f8b4bdc42e176a4ea812a43c3174e191202a79f0fc37c04558d19b695a041553"
3 changes: 2 additions & 1 deletion tests/img_toolchain/testcases/layer_runfiles.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,5 +42,6 @@ tar_entry_exists = layer.tar.gz, foo/bar/baz.exe.runfiles/_main/data/3.txt
tar_entry_type = layer.tar.gz, foo/bar/baz.exe.runfiles/_main/data/3.txt, regular
tar_entry_size = layer.tar.gz, foo/bar/baz.exe.runfiles/_main/data/3.txt, 14
layer_invariants_intact = layer.tar.gz
file_sha256 = layer_meta.json, "047d674cf55d5d5c80e917fa5847eff506614b06328e8e8d67cecaea415a4caf"
file_contains = layer_meta.json, "history"
file_sha256 = layer_meta.json, "5924cfb69df1069c1e7870c17caa9de50f31eddec545417063f95649d13035bb"
file_sha256 = layer.tar.gz, "1fce6f5736c0594d6a2273251e123babff5645f3dace6a06142d4a24a290dec7"
Loading