Skip to content

Add an API for generating content directly in the tool cache #2485

Description

@ericsciple

Describe the enhancement

@actions/tool-cache currently supports populating the tool cache with
cacheFile() and cacheDir(). Both APIs copy existing content into the cache:

  • cacheFile() copies through @actions/io, which uses Node.js
    fs.copyFile().
  • cacheDir() recursively copies each child through @actions/io.

Some callers need to generate content directly at its final cache location.
Copying is either unnecessarily expensive or changes important filesystem
properties. Examples include:

  • sparse filesystem images, where fs.copyFile() can materialize holes;
  • large generated SDK or tool directories;
  • files produced using reflinks or hardlinks;
  • transformations whose output is already being written once and should not be
    written a second time merely to enter the tool cache.

Please add a callback-based API that lets the caller populate a toolkit-managed
cache destination directly, while the toolkit continues to own the cache layout,
completion marker, cleanup, version normalization, and architecture handling.

One possible shape is:

export async function cacheGeneratedTool(
  tool: string,
  version: string,
  producer: (destination: string) => Promise<void>,
  arch?: string
): Promise<string>

Expected semantics:

  1. Normalize version and resolve arch consistently with cacheFile() and
    cacheDir().
  2. Remove an incomplete destination and its stale .complete marker.
  3. Create the destination directory in the tool cache.
  4. Invoke producer(destination).
  5. Write the existing .complete marker only after the producer resolves.
  6. If the producer rejects, remove the incomplete destination and do not leave a
    completion marker.
  7. Return the completed cache directory.
  8. Do not copy, move, archive, or otherwise reinterpret files created by the
    producer.
  9. Preserve the tool cache's current concurrency contract; this proposal does
    not require introducing cross-process locking or shared-cache coordination.

The existing cacheFile() and cacheDir() APIs should remain unchanged.
Exposing the private path and completion-marker functions directly would make it
easy for callers to forget cleanup or mark an incomplete cache entry as complete,
so a callback API appears safer.

Whether an existing completed entry should be checked by this API or by the
caller can follow the package's current conventions. For example:

let toolPath = tc.find(tool, version, arch)
if (!toolPath) {
  toolPath = await tc.cacheGeneratedTool(
    tool,
    version,
    async destination => {
      await generateToolDirectlyInto(destination)
      await validateGeneratedTool(destination)
    },
    arch
  )
}

Code snippet

A sparse-image consumer could use the API without asking the toolkit to
understand sparse files or any particular compression format:

const cached = tc.find('example-rootfs', version, arch)
if (cached) {
  return path.join(cached, 'base.ext4')
}

const cacheDir = await tc.cacheGeneratedTool(
  'example-rootfs',
  version,
  async destination => {
    const output = path.join(destination, 'base.ext4')
    await exec.exec('zstd', ['--sparse', '-dqf', compressed, '-o', output])
    await validateImage(output)
  },
  arch
)

return path.join(cacheDir, 'base.ext4')

The producer writes the output once, directly in the toolkit-managed cache
directory. Validation completes before the toolkit writes .complete.

Additional information

The motivating measurement used a 2 GiB logical sparse file with approximately
197 MiB physically allocated on ext4:

Operation Wall time Resulting allocation
Node.js fs.promises.copyFile() 2.97 seconds 2.147 GB
cp --sparse=always 0.17 seconds 206.6 MB

The exact timings depend on the host, but the semantic issue is independent of
throughput: copying through fs.copyFile() can turn a sparse cache artifact into
a fully allocated file.

The package already has the necessary internal lifecycle:
_createToolPath() prepares the destination and removes stale state, while
_completeToolPath() writes the marker consumed by find(). The proposed API
would expose that lifecycle safely without exposing those private functions or
requiring every specialized action to reproduce the tool-cache directory and
marker conventions.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions