Describe the enhancement
@actions/tool-cache currently supports populating the tool cache with
cacheFile() and cacheDir(). Both APIs copy existing content into the cache:
cacheFile() copies through @actions/io, which uses Node.js
fs.copyFile().
cacheDir() recursively copies each child through @actions/io.
Some callers need to generate content directly at its final cache location.
Copying is either unnecessarily expensive or changes important filesystem
properties. Examples include:
- sparse filesystem images, where
fs.copyFile() can materialize holes;
- large generated SDK or tool directories;
- files produced using reflinks or hardlinks;
- transformations whose output is already being written once and should not be
written a second time merely to enter the tool cache.
Please add a callback-based API that lets the caller populate a toolkit-managed
cache destination directly, while the toolkit continues to own the cache layout,
completion marker, cleanup, version normalization, and architecture handling.
One possible shape is:
export async function cacheGeneratedTool(
tool: string,
version: string,
producer: (destination: string) => Promise<void>,
arch?: string
): Promise<string>
Expected semantics:
- Normalize
version and resolve arch consistently with cacheFile() and
cacheDir().
- Remove an incomplete destination and its stale
.complete marker.
- Create the destination directory in the tool cache.
- Invoke
producer(destination).
- Write the existing
.complete marker only after the producer resolves.
- If the producer rejects, remove the incomplete destination and do not leave a
completion marker.
- Return the completed cache directory.
- Do not copy, move, archive, or otherwise reinterpret files created by the
producer.
- Preserve the tool cache's current concurrency contract; this proposal does
not require introducing cross-process locking or shared-cache coordination.
The existing cacheFile() and cacheDir() APIs should remain unchanged.
Exposing the private path and completion-marker functions directly would make it
easy for callers to forget cleanup or mark an incomplete cache entry as complete,
so a callback API appears safer.
Whether an existing completed entry should be checked by this API or by the
caller can follow the package's current conventions. For example:
let toolPath = tc.find(tool, version, arch)
if (!toolPath) {
toolPath = await tc.cacheGeneratedTool(
tool,
version,
async destination => {
await generateToolDirectlyInto(destination)
await validateGeneratedTool(destination)
},
arch
)
}
Code snippet
A sparse-image consumer could use the API without asking the toolkit to
understand sparse files or any particular compression format:
const cached = tc.find('example-rootfs', version, arch)
if (cached) {
return path.join(cached, 'base.ext4')
}
const cacheDir = await tc.cacheGeneratedTool(
'example-rootfs',
version,
async destination => {
const output = path.join(destination, 'base.ext4')
await exec.exec('zstd', ['--sparse', '-dqf', compressed, '-o', output])
await validateImage(output)
},
arch
)
return path.join(cacheDir, 'base.ext4')
The producer writes the output once, directly in the toolkit-managed cache
directory. Validation completes before the toolkit writes .complete.
Additional information
The motivating measurement used a 2 GiB logical sparse file with approximately
197 MiB physically allocated on ext4:
| Operation |
Wall time |
Resulting allocation |
Node.js fs.promises.copyFile() |
2.97 seconds |
2.147 GB |
cp --sparse=always |
0.17 seconds |
206.6 MB |
The exact timings depend on the host, but the semantic issue is independent of
throughput: copying through fs.copyFile() can turn a sparse cache artifact into
a fully allocated file.
The package already has the necessary internal lifecycle:
_createToolPath() prepares the destination and removes stale state, while
_completeToolPath() writes the marker consumed by find(). The proposed API
would expose that lifecycle safely without exposing those private functions or
requiring every specialized action to reproduce the tool-cache directory and
marker conventions.
Describe the enhancement
@actions/tool-cachecurrently supports populating the tool cache withcacheFile()andcacheDir(). Both APIs copy existing content into the cache:cacheFile()copies through@actions/io, which uses Node.jsfs.copyFile().cacheDir()recursively copies each child through@actions/io.Some callers need to generate content directly at its final cache location.
Copying is either unnecessarily expensive or changes important filesystem
properties. Examples include:
fs.copyFile()can materialize holes;written a second time merely to enter the tool cache.
Please add a callback-based API that lets the caller populate a toolkit-managed
cache destination directly, while the toolkit continues to own the cache layout,
completion marker, cleanup, version normalization, and architecture handling.
One possible shape is:
Expected semantics:
versionand resolvearchconsistently withcacheFile()andcacheDir()..completemarker.producer(destination)..completemarker only after the producer resolves.completion marker.
producer.
not require introducing cross-process locking or shared-cache coordination.
The existing
cacheFile()andcacheDir()APIs should remain unchanged.Exposing the private path and completion-marker functions directly would make it
easy for callers to forget cleanup or mark an incomplete cache entry as complete,
so a callback API appears safer.
Whether an existing completed entry should be checked by this API or by the
caller can follow the package's current conventions. For example:
Code snippet
A sparse-image consumer could use the API without asking the toolkit to
understand sparse files or any particular compression format:
The producer writes the output once, directly in the toolkit-managed cache
directory. Validation completes before the toolkit writes
.complete.Additional information
The motivating measurement used a 2 GiB logical sparse file with approximately
197 MiB physically allocated on ext4:
fs.promises.copyFile()cp --sparse=alwaysThe exact timings depend on the host, but the semantic issue is independent of
throughput: copying through
fs.copyFile()can turn a sparse cache artifact intoa fully allocated file.
The package already has the necessary internal lifecycle:
_createToolPath()prepares the destination and removes stale state, while_completeToolPath()writes the marker consumed byfind(). The proposed APIwould expose that lifecycle safely without exposing those private functions or
requiring every specialized action to reproduce the tool-cache directory and
marker conventions.