Skip to content

provenance-internals.md

Latest commit

 

History

History
531 lines (420 loc) · 25.6 KB

provenance-internals.md

File metadata and controls

531 lines (420 loc) · 25.6 KB
Raw

Provenance & manifest — producer internals

Status: implemented. This is the maintainer-facing companion to manifest.md (the consumer guide). It records the design decisions and invariants for producing provenance, the cache, the embedded bundle, and the generated headers.

Audience: gloam maintainers. If you only want to read a manifest, see manifest.md.

Goals

  1. Record the exact upstream provenance of every input that influences a generated loader: repository, commit, and content (blob) hash.
  2. Surface it in --version, in the per-file generated header block, and in .gloam/manifest.json.
  3. Make committed output byte-identical for identical inputs + gloam version, with no timestamps anywhere in committed artifacts.
  4. Scope provenance to what actually influenced each output file — not to the whole command line. This is what fixes the long-standing ANGLE misattribution bug (see below).
  5. Use one acquisition path for --fetch, the embedded bundle, and --lock.

Acquisition model: per-repo clusters, git-object walking

Upstream files are grouped into repository clusters. Provenance is resolved once per cluster and shared across every file from that cluster — this is both correct (all files in a snapshot share one commit) and cheap (it collapses ~25 files into a handful of clusters).

cluster (repo) branch files license / holder
KhronosGroup/OpenGL-Registry main xml/gl.xml, xml/glx.xml, xml/wgl.xml Apache-2.0 / The Khronos Group Inc.
KhronosGroup/EGL-Registry main api/egl.xml, api/KHR/khrplatform.h, api/EGL/eglplatform.h Apache-2.0 / The Khronos Group Inc.
KhronosGroup/Vulkan-Docs main xml/vk.xml Apache-2.0 / The Khronos Group Inc.
KhronosGroup/Vulkan-Headers main include/vulkan/vk_platform.h, include/vk_video/* Apache-2.0 / The Khronos Group Inc.
angle/angle (chromium.googlesource.com; mirror: github.com/google/angle, fallback) main scripts/gl_angle_ext.xml, scripts/egl_angle_ext.xml BSD-3-Clause / The ANGLE Project Authors
Cyan4973/xxHash dev xxhash.h BSD-2-Clause / Yann Collet
tycho/gloam-registry master xml/glsl_exts.xml MIT / Steven Noonan

Endpoints: identity vs transport

A cluster's repo/repo_url are its canonical identity — that is what every pin records. Separately, each cluster carries an ordered list of fetch endpoints (canonical first, mirrors after) that are pure transport: every per-cluster operation (HEAD resolution, directory listing, content-at-commit) tries them in order, warning and falling over on failure. Because endpoints share one git history, object SHAs are identical across them, so which endpoint served the bytes is invisible in every output (manifest, preamble, --version) and in the cache (rows are keyed by canonical slug and object SHAs).

Lag semantics: a mirror can lag the canonical repo but never diverge from it. Content fetches are pinned to the already-resolved commit, so a lagging mirror either serves byte-identical content or 404s — never wrong bytes. A 404 counts as that endpoint failing; if no endpoint has the commit, the run fails with advice to retry shortly (there is no silent HEAD re-resolution, which could tear the snapshot). The content-addressed blobs-API fallback for GC'd pinned commits is a nearly-never robustness path, not a design pillar.

Two dialects are spoken. GitHub endpoints use the REST API plus the raw host as described below. Gitiles endpoints (e.g. chromium.googlesource.com for ANGLE) use +log/refs/heads/{branch}?n=1&format=JSON for the branch tip, +/{commit}/{dir}?format=JSON for listings (entries carry git object types — files are "blob" — with the git blob SHA-1 in id), and +/{commit}/{path}?format=TEXT for content (base64 body). Gitiles specifics: every JSON response starts with a )]}' anti-XSSI line that must be stripped (its absence is a hard error), no ETag is served (HEAD resolution is always unconditional there; ETag conditionals remain GitHub-only), there is no blob-by-SHA endpoint, and no auth of any kind is sent to Gitiles hosts.

Resolving a cluster (race-free, near-zero metered calls)

For each cluster we resolve a consistent snapshot pinned to one commit:

  1. GET /repos/{o}/{r}/git/ref/heads/{branch} → tip commit SHA — sent conditionally when possible: the response ETag from the previous resolution is stored (see the http_etags table) and replayed as If-None-Match. An unchanged ref answers 304 Not Modified, which GitHub does not count against the API rate limit — a free heartbeat that refreshes the HEAD TTL. Only a 200 (ref actually moved, or no stored ETag) costs a metered call.
  2. Registries (OpenGL-Registry, EGL-Registry) and angle/angle typically have no semver tags and show a bare commit; Vulkan-Docs, Vulkan-Headers, and xxHash tag and show vX.Y.Z-N-gSHA.
  3. Per changed directory (missing files grouped by parent directory): one GET /repos/{o}/{r}/contents/{dir}?ref={commit} directory listing → the blob SHA of every file in it. Tree entries are recorded for the cluster's tracked files; a tracked file absent from its listing is a hard error. (Contents directory listings cap at 1000 entries; every tracked directory is far below that.)
  4. Per file whose blob isn't already cached: fetch the bytes from https://raw.githubusercontent.com/{o}/{r}/{commit}/{path} — the raw host is unmetered — and verify them locally by git-blob hashing against the SHA from step 3 (mismatch is a hard error). An unchanged blob at a new commit is served from the cache with zero download.

Everything is pinned to the commit from step 1, so even if upstream commits mid-resolution the content we get is exactly what our metadata describes — the metadata/data race is eliminated without the weaker "re-fetch and compare" approach. And because raw-host bytes are verified against API-reported blob SHAs, content correctness never rests on the unauthenticated raw host.

Why directory listings, not a recursive tree walk. An earlier design used git/trees/{commit}?recursive=1 to resolve paths to blobs in one call. But the recursive-tree endpoint truncates at GitHub's tree limits, and large repos (angle/angle, Vulkan-Docs) can exceed them — silently dropping the entry we need. Per-directory Contents listings never truncate for directories of this size, and one call covers every tracked file in the directory. (A still earlier iteration used one Contents call per file, with inline base64 content and a blobs-API follow-up for >1 MB files; the listing+raw split replaces both — the metered API now carries only metadata, never content.)

For --lock, the commit/blob are already known, so we skip resolution and fetch each file's content cache-first, then raw-host-first (verified against the pinned blob), with the content-addressed GET /repos/{o}/{r}/git/blobs/{blob} API as the fallback when the pinned commit is no longer reachable upstream.

--lock shortcut

When generating from a --lock manifest, commit/tree are already known. We skip steps 1–3 and fetch each required file's content directly (step 4 only), cache permitting: raw host by pinned commit + path, verified against the pinned blob, with the blobs API as fallback. Cheaper and fully reproducible.

--lock requires that the locked blobs are obtainable. The user must therefore either:

  1. pass --fetch as well (blobs are fetched by id, cache-first), or
  2. have a gloam build whose bundled files match the locked blobs.

gloam computes the required files for the run and compares each one's pinned blob against the corresponding entry in bundled/provenance.json. If every required blob matches the bundle, --lock works offline against bundled content. If any differs or is absent, gloam refuses and instructs the user to add --fetch — bundled content cannot satisfy that lock.

GITHUB_TOKEN

Calls to api.github.com read $GITHUB_TOKEN when present and fall back to unauthenticated access otherwise. The token is only consulted for api.github.com requests — it is never sent to raw.githubusercontent.com (the raw host serves public content, and the API credential must not leak cross-host).

  • Unauthenticated: 60 req/hr — comfortable for --fetch now that content is unmetered, 304s are free, and only ref moves + directory listings are billed.
  • Authenticated: 5000 req/hr for a PAT; 1000 req/hr/repo for the Actions GITHUB_TOKEN. Reading other public repos' APIs is allowed for any token.

In gloam-pregen CI, pass the automatic token — no PAT to mint or store:

- name: Generate …
  env:
    GITHUB_TOKEN: ${{ github.token }}
  run: gloam --api … --fetch …

Cache

Reduce API traffic and make --fetch pleasant offline-ish. The cache exists only to serve --fetch; bundled-mode generation reads embedded files directly and never touches it. Accordingly the cache implementation (rusqlite/bundled SQLite) lives behind the existing fetch feature, so default / --no-default-features builds stay lean and SQLite-free.

Location via the directories crate's cache dir:

  • Windows: %LOCALAPPDATA%\gloam\cache
  • Linux: ~/.cache/gloam
  • macOS: ~/Library/Caches/gloam

Why SQLite, not a snapshot tarball

The first design (a .tar snapshot of HEAD + JSON sidecars) only modeled repeated --fetch against current HEAD. --lock breaks that assumption: it can pin older commits/blobs, and using an old lock must not clobber our notion of HEAD. We need to cache non-HEAD content alongside HEAD, and evict by use, not by recency of HEAD. That calls for two independent TTL classes and transactional eviction — a relational store fits far better than a tar we'd have to rewrite wholesale and GC by hand.

The cache is a single SQLite file (still honoring the single-file constraint), in WAL mode for safe concurrent access.

Backend choice: rusqlite (bundled SQLite)

We use rusqlite with its bundled feature, behind the fetch feature. Rationale:

  • No new toolchain cost. The fetch feature already pulls ureq + rustls, whose crypto provider (ring) compiles C/assembly — so --features fetch already requires a C compiler. Bundled SQLite adds no new category of dependency, only ~10–30s one-time compile and ~1 MB, both fetch-only. (Default --no-default-features builds have neither fetch nor SQLite.)
  • Mature + inspectable + relational. The data model is genuinely relational (many-to-many blob↔commit, joins, range-based eviction); SQL expresses it cleanly, and the file is inspectable with stock sqlite3 / DB Browser.

Pure-Rust SQL alternatives were considered and rejected: Turso/Limbo (pure-Rust, SQLite-file-compatible, but beta and churning) and GlueSQL (pure-Rust but early-stage and not SQLite-file-compatible, losing third-party inspectability). Their only advantage — avoiding cc — is already moot given the TLS stack.

Integrity note: blobs are keyed by SHA, so any blob read from the cache is re-hashed against its key before use; a mismatch is treated as a miss and re-fetched. Content correctness never depends on the storage engine being bug-free.

Schema

-- HEAD tracking per repo; drives HEAD TTL (re-resolution), NOT eviction.
repos(repo TEXT PRIMARY KEY, branch TEXT, head_commit TEXT, head_fetched_at INTEGER)

-- Commit rows exist for eviction bookkeeping (orphaned tree entries are
-- swept when their commit is evicted).
commits(commit_sha TEXT PRIMARY KEY, repo TEXT, last_used INTEGER)

-- Many-to-many bridge: a file path at a commit resolves to a blob.
tree_entries(commit_sha TEXT, path_in_repo TEXT, blob_sha TEXT,
             PRIMARY KEY (commit_sha, path_in_repo))

-- Content, deduped by SHA. NO commit column: a blob lives in many commits.
blobs(blob_sha TEXT PRIMARY KEY, content BLOB, last_used INTEGER)

-- Response ETags for conditional requests, keyed by the full request URL
-- (an ETag belongs to a *request*, not to a domain object). Stores NO
-- payloads: a 304 means "whatever the caller derived from this URL last
-- time still holds", and the caller refreshes its own domain row (e.g.
-- repos.head_fetched_at). Rows are tiny and never evicted.
http_etags(url TEXT PRIMARY KEY, etag TEXT, fetched_at INTEGER)

Two TTL classes

  • HEAD TTL (repos.head_fetched_at, default ~1 day): how long a cached HEAD is trusted before a no-lock --fetch re-resolves it. Governs re-resolution only — a stale HEAD is never evicted, just refreshed on next use.
  • Object TTL (commits.last_used, blobs.last_used, default longer): how long unused commits/blobs survive before eviction. Refreshed (last_used = now) on every reuse.

Access patterns

  • --fetch, no lock: if repos.head_fetched_at is fresh, reuse head_commit → tree_entries → blobs with zero API calls. If stale, re-resolve HEAD conditionally with the stored ETag (acquisition model); an unchanged HEAD answers 304 (unmetered) and just bumps head_fetched_at, a moved HEAD inserts the new commit/tree/blobs and records the new ETag. Old blobs stay until object-TTL eviction.
  • --fetch --lock: never touches repos (HEAD). Keys off the manifest's pinned commit/blob SHAs: serve blobs straight from blobs, fetch missing ones by id and insert them, and opportunistically record the manifest's commit/tree rows. Bumps last_used on everything reused.

Seeding from the bundle (unifies the lookup path)

In fetch builds, on each run gloam seeds the cache from the embedded bundle if the content isn't already present: INSERT OR IGNORE the bundled blobs plus their commits/tree_entries rows derived from bundled/provenance.json. This collapses the lookup to a single path — "in cache? → else fetch (if --fetch) → else error" — with no separate "is it bundled?" branch, because bundled content is in the cache.

  • Seeding writes commits/tree_entries/blobs but not repos.head_* — the bundled commit isn't necessarily HEAD, so seeding behaves like a lock, not a HEAD refresh.
  • Seeded blobs are ordinary cache entries (evictable by object-TTL). If one ages out, the next run that needs it re-seeds for free from the embedded bytes — idempotent and self-healing. INSERT OR IGNORE keyed on blob SHA makes re-seeding a no-op when present.

Build-configuration split:

  • fetch ON: cache exists; seed-from-bundle applies; lookups are cache-or-fetch as above.
  • fetch OFF (--no-default-features): no SQLite, no cache, no network. Bundled files are read directly. --lock still works only if its pinned blobs match the bundled blobs (compare lock pins vs bundled/provenance.json, both in-binary); otherwise it errors that this build has no --fetch.

Self-sufficiency invariant. The cache rows carry everything outputs need (repo, commit, blob, path), so a run where every required file hits the cache emits full headers and .gloam/manifest.json with zero network access. The cache is always sufficient to regenerate all provenance metadata for the content it holds.

Schema versioning

The cache is pure derived data — fully rebuildable from the network and the bundle — so there is nothing to migrate and no data to lose. We therefore do not version tables or write migrations. Instead, the schema version is stored in the file header via PRAGMA user_version; on startup, if it doesn't match gloam's expected version, the database is dropped and recreated from scratch. (Versioned table names risk orphaning old tables; migrations add unwarranted complexity.)

Eviction / GC

Cheap SQL, no manual walking:

DELETE FROM commits WHERE last_used + :object_ttl < :now;
DELETE FROM blobs   WHERE last_used + :object_ttl < :now;
-- then sweep orphaned tree_entries (or rely on ON DELETE CASCADE)

A blob is evicted purely by its own last-use, independent of any commit — correct, since the same blob is shared across commits that didn't modify it.

Subset projection

A cache, bundle, or --lock manifest may hold more than a given run needs. We project provenance down to the files the run's outputs actually use, by the same rules that select which copyright notices to display. One projection function feeds the header, --version, and the output manifest.

Embedded bundle

bundled/ keeps its file tree and gains a checked-in bundled/provenance.json — the canonical, human-readable, deterministic pin set — embedded via include_str! and parsed at startup. (The cache is a private SQLite DB, a separate concern; bundled/provenance.json is the checked-in, reviewable artifact.) The bundle has no TTL; it is refreshed only when a maintainer re-bundles.

bundled/provenance.json is the source of truth for --version and for bundled (non---fetch) generation. Content hashes can be recomputed from the embedded bytes at build time to detect drift between the manifest and the files.

cargo xtask bundler (replaces scripts/fetch_bundled.sh)

A dev-only workspace member that runs the same acquisition path as --fetch and writes both the bundled/ files and bundled/provenance.json. This removes the shell script and guarantees bundled and fetched provenance are produced identically. The reachability test currently in src/fetch.rs is reworked to exercise the API path.

Structural prerequisite. To let xtask call gloam's acquisition code, the repo becomes a cargo workspace and gloam gains a library target (src/lib.rs) exposing the acquisition/provenance modules; xtask depends on gloam as a lib. This does not change how the gloam binary builds or publishes (xtask is not a dependency of gloam), and the lib target also makes the acquisition + cache code directly unit-testable. Invoked via a .cargo/config.toml alias: cargo xtask bundle. scripts/fetch_bundled.sh is retained as a thin wrapper that invokes it.

gloam lock — manifest-only snapshot subcommand

gloam lock --out <file> writes a provenance-only manifest (no loader output; an empty output BOM) pinning every supported upstream source — the reusable point-in-time lock referenced in the consumer guide. It shares the loader path with --fetch: bundled provenance by default, or upstream HEAD with --fetch (as in the pregen preflight). The name mirrors the --lock input flag: gloam lock produces what --lock consumes.

Today the snapshot always covers all sources — a safe superset for any subset --lock. Per---api narrowing is a possible future refinement.

Output scoping: per-FeatureSet, and the ANGLE bug

The bug. Today the ANGLE notice is keyed on the spec family (src/preamble.rs: matches!(fs.spec_name, "gl" | "egl")) and hardcodes the string "(gl_angle_ext.xml, egl_angle_ext.xml)". So a gl:core loader wrongly advertises ANGLE — and names an EGL file in a GL loader — even though no ANGLE extension is in scope. The parser also flattens all docs into one RawSpec, losing which file contributed what.

The model. The provenance unit is the generated output file, i.e. one per FeatureSet. A merged gl:core,gles2,egl run still emits separate gl.{h,c} and egl.{h,c}. Each FeatureSet carries the set of source files that fed it:

contributing sources(FeatureSet) =
      primary spec XML for this FeatureSet
    + supplemental XMLs merged into this FeatureSet
    + required_headers emitted for this FeatureSet

This is coarse (file granularity, not per-symbol): a file counts if it was merged in, even if its content was fully redundant. The header, copyright grouping, and output[].derived_from all derive from this one per-FeatureSet set, so they agree by construction.

Request-aware supplemental merge. To make gl:core exclude gl_angle_ext.xml, the supplemental-merge decision must depend on the requested APIs, not just the spec family (gl:core and gles2 share spec_name == "gl"). The merge mapping:

supplemental merged when…
glsl_exts.xml any GL-family API (gl/gles1/gles2) — it carries supported="gl" and supported="gl|gles2" entries
gl_angle_ext.xml a GLES API (gles1/gles2) is requested
egl_angle_ext.xml egl is requested

So gl:core alone merges gl.xml + glsl_exts.xml only (plus its headers); no ANGLE. The production gl:core,gles2,egl --merge build merges gl_angle_ext.xml into the GL loader (GLES2 in scope) and egl_angle_ext.xml into the EGL loader. This is low-risk for output bytes — ANGLE's GLES-targeted extensions would not be selected into a core-only loader anyway — but validate the merged production output stays byte-identical when implementing.

This threads the requested API set into fetch::load_spec (today it only knows spec_name).

Attribution: static table × fetched facts

Keep fetched provenance purely factual (repo, commit, blob). Copyright text — holder, year range, license — is gloam's own static knowledge, a table keyed by cluster. Combine the two at emit time. Years come from build_info::BUILD_YEAR (deterministic per gloam release), as today.

This also closes a pre-existing gap: xxHash currently has no attribution even though xxhash.h ships in output. Add its BSD-2-Clause notice, scoped (like every notice) to loaders that actually emit the file.

Deterministic ordering (must be stable)

  • Copyright notices: gloam's MIT notice first; then one notice per distinct (holder, license) among contributing sources, sorted by SPDX identifier, then holder. (⇒ Apache-2.0 Khronos, BSD-2-Clause xxHash, BSD-3-Clause ANGLE.)
  • Sources block: repositories sorted case-insensitively by owner/name slug; files within a repo sorted by path_in_repo.
  • provenance object: keyed by logical file name, keys sorted ascending (use IndexMap populated in sorted order to preserve it through serde).
  • output array: sorted by path ascending.

No HashMap iteration order, no timestamps, no randomness — per the project's byte-identical-output rule.

Header block assembly

Order within the comment: Generated by … {describe} → reproducing command line → Extensions: summary (existing logic) → copyright notices → Generated from the following upstream sources: (repo-grouped, two-level) → DO NOT EDIT footer. The two-level source layout (repo+commit on one line, indented path (blob …) lines under it) keeps a repo-wide commit bump from churning every file line in downstream diffs.

DO NOT EDIT moves to its own footer:

DO NOT EDIT. This file is generated by gloam and will be overwritten;
make changes to the gloam invocation or inputs, not to this file.

--version output

--version reports the embedded bundle's provenance (from bundled/provenance.json) using the same repo-grouped layout as the header sources block — repo + git describe per line, indented path (blob …) lines beneath — minus the C-comment asterisks. There are few enough files that listing them all is reasonable; a compact per-repo-only mode can be added later if it grows unwieldy. The formatter is shared with the header builder so the two never drift.

.gloam/manifest.json

Always written to the output root, pretty-printed (line-diffable). Three sections — gloam, provenance, output — fully specified in manifest.md. Producer notes:

  • provenance is the pin set, not the union of usage. On a fresh run it is exactly what was fetched; under --lock it is the input manifest's provenance carried verbatim (no add/remove), even if this run used a subset.
  • output[].derived_from is the per-file usage (the contributing-sources set of that file's FeatureSet, or the single source file for a verbatim header).
  • output[].blob and the upstream blob fields are git blob SHA-1s (git hash-object), reproducible and verifiable with stock git.
  • --lock reads only provenance; gloam and output are regenerated.

gloam-pregen workflow changes

These are gloam-pregen process details, not main-repo behavior.

  • Add GITHUB_TOKEN: ${{ github.token }} to each generation step's env.
  • Snapshot preflight. Run the manifest-only subcommand once at the top of the job (with --fetch) to write a manifest.json at the pregen repo root pinning every repo at one consistent instant. Then drive all three loaders (gl-egl, vulkan, vulkan-aio) with --lock manifest.json --fetch. Upstream HEAD is resolved exactly once; the three loaders are provably generated from the same snapshot; subsequent blob fetches hit the cache.
    • Consequence (accepted): each loader's .gloam/manifest.json carries the full snapshot pin set verbatim, so e.g. vulkan/.gloam/manifest.json lists gl/egl/angle pins it never referenced. This is the intended "broad lock, trimmed output → unreferenced pins" behavior.
  • Replace the weak commit message. Instead of a second gloam invocation (which would race the first), the commit step reads the just-written .gloam/manifest.json from each output tree and formats a human-readable message body from gloam + provenance (gloam version, and each upstream repo's describe/commit that changed). The manifest is committed alongside the loaders, so its diff is itself an auditable record.
  • Because committed artifacts have no timestamps, the daily job only produces a diff (and thus a commit) when upstream content actually changed.

Schema versioning

schema_version starts at 1. Bump on incompatible layout changes; additive backward-compatible fields need no bump. Consumers reject unknown majors.