Build definition API checklist

[nalind]

We'd like to be able to drive a build based on already-parsed build definition, but being able to consume those specifications properly is going to require that we add to our lower-level APIs support for a number of features which can be expressed in a definition that the APIs can't currently fulfill. It also makes sense to expose some of these features through the corresponding CLI commands.

• Op_Exec.Meta.ValidExitCodes:

  • Add RunOptions.ValidExitCodes (type: []int32) // expect/accept any of these exit status values as success instead of 0
  • Add --valid-exit-codes to buildah run, default []int32{0}.
  • Add an integration test using buildah run.

• Op_Exec.Mounts.CacheOpt.Sharing:

  • Reported in Unsupported RUN --mount=…,sharing=private podman#27188
  • Teach internal.volumes.GetCacheMount() about "sharing=private"
  • Add an integration test using buildah run.

• Op_Exec.Security:

  • Think really hard about ways to avoid having to do this one.
  • Maybe come back to this one later.

• Op_File.Actions.FileAction_Copy.FollowSymlink:

  • Add AddAndCopyOptions.FollowSymlink (type: go.podman.io/image/v5/types.OptionalBool) // follow symlinks when reading globbed items, default true
  • Use to set or clear copier.GetOptions.NoDerefSymlinks

• Op_File.Actions.FileAction_Copy.DirCopyContents:

  • Add AddAndCopyOptions.DirCopyContents (type: go.podman.io/image/v5/types.OptionalBool) // copy the directory's contents instead of the directory with its contents below it, default true
  • Use to set or clear copier.GetOptions.KeepDirectoryNames

• Op_File.Actions.FileAction_Copy.AttemptUnpackDockerCompatibility:

  • Add copier.PutOptions.AttemptUnpackDockerCompatibility (type: bool) // detect if globbed items are possibly-compressed archives, and if so, extract them instead of copying them
  • Add to unit tests in copier package

• Op_File.Actions.FileAction_Copy.CreateDestPath:

  • Add copier.CreateDestPath (type: go.podman.io/image/v5/types.OptionalBool) // enable or disable the implicit creation of the destination directory if it don't exist yet
  • Add to unit tests in copier package

• Op_File.Actions.FileAction_Copy.AllowWildcard:

  • Add AddAndCopyOptions.AllowWildcard (type: go.podman.io/image/v5/types.OptionalBool) // glob for source locations, default true
  • Add copier.StatOptions.AllowWildcard
  • Add copier.GetOptions.AllowWildcard
  • Add to unit tests in copier package
  • Add --allow-wildcard to buildah add and buildah copy, default true.
  • Add integration tests

• Op_File.Actions.FileAction_Copy.AllowEmptyWildcard:

  • Add AddAndCopyOptions.AllowEmptyWildcard (type: go.podman.io/image/v5/types.OptionalBool) // don't return an error if globbing for source locations matches nothing, default true
  • Add copier.StatOptions.AllowEmptyWildcard
  • Add copier.GetOptions.AllowEmptyWildcard
  • Add to unit tests in copier package
  • Add --allow-empty-wildcard to buildah add and buildah copy, default false.
  • Add integration tests

• Op_File.Actions.FileAction_Copy.Timestamp:

  • Add copier.PutOptions.Timestamp
  • Add to unit tests in copier package

• Op_File.Actions.FileAction_Copy.IncludePatterns:

  • Add AddAndCopyOptions.Includes (type: []string) // complement to AddAndCopyOptions.Excludes
  • Add --include to buildah add and buildah copy
  • Add integration tests, including testing with both of --include and --exclude at the same time

• Op_File.Actions.FileAction_Copy.RequiredPaths:

  • Add AddAndCopyOptions.RequiredPaths (type: []string) // names we require to be in the set of things copied (as a safety check?)
  • Add --required-path to buildah add and buildah copy
  • Add integration tests, including testing that --required-path without --include fails

• Op_File.Actions.FileAction_Copy.ModeStr:

  • This is Error parsing chmod +x in ADD/COPY instructions podman#28293, which suggests using https://github.com/tonistiigi/dchapes-mode
  • Add AddAndCopyOptions.ChmodStr (type: string) // non-numeric argument that "chmod(1)" would understand, overrides Chmod field
  • Add copier.GetOptions.ChmodStr
  • Add copier.PutOptions.ChmodStr
  • Add unit tests to the copier package
  • Detect string values for --chmod in buildah add and buildah copy
  • Add a conformance test and/or integration tests

• Op_File.Actions.FileAction_Mkfile:

  • Add copier.Mkfile(), possibly just creating an in-memory buffer and feeding it to copier.Put()
  • Add unit test to the copier package

• Op_File.Actions.FileAction_Mkdir.MakeParents:

  • Add copier.MkdirOptions.MakeParents (type: go.podman.io/image/v5/types.OptionalBool) // default true
  • Add unit test to the copier package

• Op_File.Actions.FileAction_Rm.AllowWildcard:

  • Add AllowWildcard to copier.RemoveOptions (type: bool) // glob for items to remove, default false
  • Add unit test to the copier package

• Op_File.Actions.FileAction_Rm.AllowNotFound:

  • Add AllowNotFound to copier.RemoveOptions (type: bool) // don't return an error if item is already not present
  • Add unit test to the copier package

• Op_File.Actions.FileAction_Symlink:

  • Add copier.Symlink(), possibly just creating an in-memory buffer and feeding it to copier.Put()
  • Add unit test to the copier package

General catchup:

• Add fields to copier.StatForItem for representing device/socket/fifo types, major/minor device numbers, and xattrs, and populate them

[nalind]

(Adding as a comment so that it's easier for folks to follow up.)

One thing we're going to need that is larger than an API flag is better support for Exec ops. The current Builder.Run() implementation is a method on an object that we initialize using defaults or an image, and since a definition supplies us with the configuration blob for the final result but none of the intermediates, it's not something that's easily mapped to the current Builder object life cycle, where an image build can commit after each step.
To that end, I think we'll need to factor most of the current Run() method into a standalone function that can accept all of the current options Run() accepts, and also have all of the state that the current implementation pulls from the Builder object supplied as part of the function call. The Run() method uses a number of helper methods, so if it's more efficient to turn the bulk of Run() and those helper methods into a smaller "runner" object instead of a function and helper functions, and have the Run() method instantiate that object using state information from the Builder to continue to provide the functionality that it does now, then that's the way to go.