Skip to content

CONTRIBUTING.md

Latest commit

 

History

History
260 lines (211 loc) · 12 KB

CONTRIBUTING.md

File metadata and controls

260 lines (211 loc) · 12 KB
Raw

Contributing to gloam

Design specs: provenance tracking and the generated .gloam/manifest.json are specified in docs/manifest.md (consumer-facing format) and docs/provenance-internals.md (producer internals: acquisition, cache, bundler, header layout).

Architecture overview

The pipeline is strictly linear: parse -> IR -> resolve -> generate.

CLI args
  -> fetch/bundled XML specs
  -> parse into RawSpec (IR)
  -> resolve into FeatureSet (indexed, sorted, grouped)
  -> generate C code via minijinja templates (or Rust via string emission)

Module map

Module Purpose
main.rs Entry point; orchestrates CLI -> resolve -> generate
cli.rs clap-derived CLI definitions, ApiRequest parsing, extension filter parsing
ir.rs Raw intermediate representation types directly from XML (pre-resolution)
bundled.rs Compile-time-embedded XML specs and auxiliary headers (include_str!)
fetch.rs Load specs from bundled copies or remote Khronos URLs (--fetch)
build_info.rs Git version metadata embedded at compile time (generated by build.rs)
preamble.rs Copyright/license/provenance comment block for generated files
regen.rs gloam regen: replays each tree's recorded command line to regenerate it in place
parse/ XML -> RawSpec IR
parse/mod.rs Orchestrator; raw C text extraction, enum value computation
parse/types.rs <types> -> RawType[] with topological sort (Kahn's algorithm)
parse/enums.rs <enums> -> flat enums + Vulkan typed enum groups
parse/commands.rs <commands> -> RawCommand[] with alias-chain fixup and Vulkan scope inference
parse/features.rs <feature> + <extension> -> RawFeature[] + RawExtension[]
resolve/ RawSpec + CLI args -> FeatureSet
resolve/mod.rs Three-phase orchestrator (selection -> materialization -> grouping)
resolve/types.rs Public output types: FeatureSet, Feature, Extension, Command, etc.
resolve/selection.rs Which features/extensions are "in" (filter, promoted, predecessors)
resolve/requirements.rs Collects required types/enums/commands from selected features+extensions
resolve/commands.rs Indexed Command entries, PFN ordering optimization, alias pair extraction
resolve/pfn.rs PFN range tables (feature -> command index ranges)
resolve/enums.rs Flat enums and Vulkan enum groups for the resolved set
resolve/typedefs.rs Type list with dependency ordering and include-guard inference
resolve/protect.rs Platform protection lattice; group-by-protection coalescing
resolve/spec_info.rs Spec-level constants: display names, PFN prefixes, name prefixes
generator/ FeatureSet -> output files
generator/c/mod.rs C generator: template rendering, function-name blob layout, aux header copying
generator/c/templates/ Minijinja templates (header.h.j2, source.c.j2, hash_search.j2, library.j2, loader.j2, etc.)
generator/rust/mod.rs Rust generator (experimental; GL/GLES): emits a self-contained #![no_std] crate by string emission

Key data types

  • RawSpec (ir.rs): Everything parsed from one XML spec family (types, enums, commands, features, extensions). This is the pre-resolution IR.
  • FeatureSet (resolve/types.rs): Fully indexed, sorted, protection-grouped output ready for template rendering. Contains features, extensions, commands, types, flat enums, enum groups, PFN range tables, extension index subsets, and protection-grouped lists.
  • Command (resolve/types.rs): Indexed command with short name, PFN type name, parameter string, scope (for Vulkan), and protection.
  • Protection (resolve/protect.rs): Unconditional or Guarded(macros) — used to emit #ifdef blocks in generated code.

Resolution phases

  1. Selection: Determine which API features and extensions are included based on CLI flags (--api, --extensions, --promoted, --predecessors, --baseline).
  2. Materialization: Build indexed arrays of commands, types, enums. Optimize command ordering to minimize PFN range fragmentation. Build PFN range tables.
  3. Grouping: Coalesce items by platform protection for #ifdef-correct header emission.

Extension detection strategy

At load time, the generated code detects driver-supported extensions without any string comparisons:

  1. Calls glGetIntegerv(GL_NUM_EXTENSIONS, &n) to get the count.
  2. Calls glGetStringi(GL_EXTENSIONS, i) for each i, hashes each name with XXH3-64 (the same algorithm used at generator time), and stores the hashes in a heap-allocated uint64_t[].
  3. Shellsorts the array in-place (Ciura gap sequence — no extra memory, ~160 bytes of code).
  4. Binary-searches the sorted driver hashes against the pre-baked known extension hash table embedded in the generated source.

This gives O(n log n) total work to detect all extensions, with O(log n) per lookup.

Alias resolution

When --alias is passed, the generated loader emits a runtime resolver: after loading all function pointers, if the canonical slot for an alias pair is null but the alias slot was loaded by the driver (or vice versa), the loaded pointer is propagated to both slots. This handles the case where a driver only exposes one spelling of a promoted function.

--alias is a runtime concern — it does not affect which extensions are selected. For selection-time alias expansion see --promoted and --predecessors.

Generated code internals

Function name blob

All function names are packed into a single kFnNameData[] string table with pre-computed offsets in kFnNameOffsets[]. This is indexed in lockstep with pfnArray[] so that loading code can look up the name for any function pointer slot by index.

PFN range tables

Features and extensions map to contiguous ranges of command indices. The tables are contiguous-run compressed: each entry is a (start_index, count) pair. This allows bulk-loading all function pointers for a feature or extension with a single loop over the range.

Platform guards

Extensions that require platform-specific headers are wrapped in #ifdef blocks (e.g. VK_USE_PLATFORM_WIN32_KHR). The resolver groups consecutive items with identical protection into ProtectedGroup<T> to minimize the number of #ifdef/#endif pairs in the generated output.

Khronos XML spec gotchas

The XML specs have numerous inconsistencies that the parser handles explicitly. These are documented in code comments as "Spec gotcha #N":

  1. Command alias entries lack full prototypes — walk alias chains to copy signatures
  2. Type and enum dependency ordering requires topological sort
  3. Vulkan enum groups can be empty after filtering — prune them
  4. Bitwidth=64 must propagate through enum alias chains
  5. GL has auto-excluded types that should be silently skipped
  6. Some XML contains C++ // comments — rewrite to /* */ for C99
  7. macOS needs a special ptrdiff_t guard for GL pointer-sized types
  8. GLX_SGIX_video_source and GLX_SGIX_dmbuffer are broken — silently drop
  9. WGL_ARB_extensions_string is mandatory but might be missing — warn
  10. Vulkan <enums type="enum"|"bitmask"> should not be re-processed as flat constants
  11. Vulkan command scope is inferred from first parameter type (Global/Instance/Device)
  12. Duplicate enum names with conflicting values must be detected and rejected

Testing

Integration tests live in tests/. They invoke the binary via assert_cmd, generate into temp directories, and optionally compile the output with cc when available. Shared helpers (the gloam runner, compile check, header/source readers) live in tests/common/mod.rs.

Test file Coverage
generate_c.rs GL/GLES C loader generation + compilation
generate_rust.rs Rust loader crate structure, constant typing, cargo check
generate_vulkan.rs Vulkan-specific generation
generate_wgl_glx.rs WGL, GLX, and cross-API edge cases
predecessor_promoted.rs --promoted and --predecessors flag behavior
provenance_manifest.rs Implicit provenance baseline on regeneration
integration_coverage.rs Determinism, exclusions, baselines, flag matrix
golden.rs Byte-exact snapshots of generated output (see below)

Run the full suite:

cargo test

Golden snapshots

tests/golden.rs compares generated .h/.c output (preamble stripped) byte-for-byte against checked-in snapshots under tests/golden/, one config per spec family plus the merged GL+GLES2 build, plus two Rust-backend configs (Cargo.toml + src/lib.rs). A refactor that should not change output is proven neutral by cargo test. When output changes deliberately — or a bundle refresh changes resolved content — re-bless and review the diff:

GLOAM_BLESS=1 cargo test --test golden
git diff tests/golden/

Miri

The library unit tests run clean under Miri and should stay that way:

cargo +nightly miri test --no-default-features --lib

Expect the first run to be slow (Miri builds its own sysroot). Scope is --lib only: the integration tests spawn the gloam binary as a subprocess, which Miri cannot interpret. Note this checks gloam itself, not the generated loaders — their dispatch paths call driver FFI, which no interpreter can execute; the generated crate's soundness-sensitive choices (Option transmutes, explicit unsafe blocks, checked dispatch) are enforced structurally at generation time instead.

Regenerating downstream trees

gloam regen [paths...] regenerates existing output trees in place by replaying the command line recorded in each tree's .gloam/manifest.json. A path may be a tree root, a directory to search recursively (the default is .), or a manifest file itself. The effective output path is derived from each manifest's own location — the recorded --out-path was relative to the original invocation's cwd, which is unknowable later — and the recorded command line is re-recorded verbatim, so regeneration works from any directory and never rewrites what the manifest says produced the tree.

By default each replay is pinned to the tree's recorded provenance (--lock semantics), so git diff shows only the effect of gloam code changes; pass --fresh to re-resolve sources instead (advancing the tree to the current bundle, or to upstream HEAD if the recorded command used --fetch).

cargo xtask regen [tree-root] [--fresh] builds the working-copy gloam and runs it over tree-root (default: the current directory) — use this for examples/ here or a gloam-pregen checkout. To review a regen, filter out gloam's own version/commit stamp churn:

git diff -I'^ \* @generated by gloam ' -I'^// @generated by gloam ' \
         -I'^    "(version|describe|commit)": ' -I'^      "blob": '

(Ignoring 6-space "blob" lines is safe: a provenance pin's blob never moves without its unfiltered sibling "commit" line moving too, so real source changes always stay visible; the output BOM's blob lines just track stamp churn.)

Debug tracing

Set GLOAM_DEBUG=1 to trace provenance/fetch activity on stderr: every HTTP request (URL, status, latency), every load::resolve call (mode, key count, duration), and engine construction / bundle-seed timings. Independent of --quiet. Use this to answer "when and why is gloam hitting the network or the cache?"

Style notes

  • Determinism: IndexMap is used throughout for insertion-order preservation. Never introduce HashMap iteration or other sources of non-determinism.
  • Self-contained binary: XML specs and auxiliary headers are embedded via include_str! from bundled/.
  • Progress output: goes to stderr, respects --quiet. Only errors use eprintln! unconditionally.
  • Preamble: generated files include the exact command line, gloam version, extension selection summary, and license notices.
  • Minimal changes: prefer small, focused changes. Don't over-abstract or add speculative infrastructure.