Skip to content

Commit 11d05ff

Browse files
committed
feat(gnenerator): add Rust backend implementation
This is a work-in-progress implementation of a rust output. Signed-off-by: Steven Noonan <steven@uplinklabs.net>
1 parent d4a293b commit 11d05ff

47 files changed

Lines changed: 105600 additions & 58 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

CONTRIBUTING.md

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -157,6 +157,7 @@ check, header/source readers) live in `tests/common/mod.rs`.
157157
| Test file | Coverage |
158158
|---|---|
159159
| `generate_c.rs` | GL/GLES C loader generation + compilation |
160+
| `generate_rust.rs` | Rust loader crate structure, constant typing, `cargo check` |
160161
| `generate_vulkan.rs` | Vulkan-specific generation |
161162
| `generate_wgl_glx.rs` | WGL, GLX, and cross-API edge cases |
162163
| `predecessor_promoted.rs` | `--promoted` and `--predecessors` flag behavior |
@@ -174,7 +175,8 @@ cargo test
174175

175176
`tests/golden.rs` compares generated `.h`/`.c` output (preamble stripped)
176177
byte-for-byte against checked-in snapshots under `tests/golden/`, one config
177-
per spec family plus the merged GL+GLES2 build. A refactor that should not
178+
per spec family plus the merged GL+GLES2 build, plus two Rust-backend
179+
configs (`Cargo.toml` + `src/lib.rs`). A refactor that should not
178180
change output is proven neutral by `cargo test`. When output changes
179181
deliberately — or a bundle refresh changes resolved content — re-bless and
180182
review the diff:
@@ -184,6 +186,22 @@ GLOAM_BLESS=1 cargo test --test golden
184186
git diff tests/golden/
185187
```
186188

189+
### Miri
190+
191+
The library unit tests run clean under Miri and should stay that way:
192+
193+
```sh
194+
cargo +nightly miri test --no-default-features --lib
195+
```
196+
197+
Expect the first run to be slow (Miri builds its own sysroot). Scope is
198+
`--lib` only: the integration tests spawn the gloam binary as a
199+
subprocess, which Miri cannot interpret. Note this checks gloam itself,
200+
not the generated loaders — their dispatch paths call driver FFI, which
201+
no interpreter can execute; the generated crate's soundness-sensitive
202+
choices (Option<fn> transmutes, explicit unsafe blocks, checked dispatch)
203+
are enforced structurally at generation time instead.
204+
187205
### Regenerating downstream trees
188206

189207
`cargo xtask regen <tree-root>` re-runs the command line recorded in every

Cargo.lock

Lines changed: 17 additions & 17 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/rust-backend.md

Lines changed: 249 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,249 @@
1+
# Rust generator backend — design notes
2+
3+
> **Status:** implemented (GL/GLES) and proven. The backend lives in
4+
> `src/generator/rust/mod.rs` (`gloam … rust`); the design below is largely
5+
> realized. It emits a self-contained `#![no_std]` crate, detects extensions
6+
> with XXH3 (reusing the resolver's `Extension.hash`), and drives a real
7+
> triangle on hardware — see
8+
> [examples/rust/gl-triangle](../examples/rust/gl-triangle/). Size is at
9+
> parity with the C loader (below). mx-global is implemented as `--mx-global`
10+
> (free-function dispatch over a write-once `UnsafeCell` global — see the
11+
> note below on why not `OnceLock`). Vulkan is still out of scope.
12+
13+
## Goal & scope
14+
15+
Add a **Rust generator backend** alongside the C backend, emitting a loader
16+
that is **more idiomatic than GLAD2's Rust output** without giving up gloam's
17+
small-code / fast-load characteristics.
18+
19+
- **GL / GLES first**, targeting a Rust port of the `gl-triangle` example.
20+
- **Vulkan later** — its type system is the hard part (see
21+
[type translation](#type-translation-the-main-new-work)); GL is much easier.
22+
23+
## What to avoid (the GLAD2 reference)
24+
25+
A GLAD2 Rust loader (`glad --api gl:core,gles2 --merge rust --alias`) was
26+
examined at `D:\dev\vk-api-loader-shootout\tmp` as the anti-pattern. Its
27+
deficiencies (all of which gloam structurally avoids):
28+
29+
- **Emits the entire registry** (every vendor extension) → ~15k lines, ~5,400
30+
flat `pub const` enums. Its monolithic `load()` touches every function, so the
31+
whole registry is **pinned against dead-code elimination** once you load.
32+
- **Every function is a `static mut FnPtr { ptr, is_loaded: bool }`** — 16 bytes
33+
each (double a raw pointer), thousands of them, as global mutable state
34+
(unsound under edition 2024).
35+
- **transmute-per-call** through the global storage; unloaded slots point at a
36+
`not_initialized` panic stub.
37+
- **No selection, no packed name blob, no bulk/range loading** (one
38+
`GetProcAddress` per function by name), **no extension hashing** (string
39+
compares). Release `.rlib` ≈ 7.7 MiB (an rlib overstates linked size, but the
40+
DCE-pinning above means the effective bloat is real).
41+
42+
## API shape (decisions)
43+
44+
### Naming — keep GL short names, own the context
45+
46+
- The generated type is **`Gl`** (UpperCamelCase — no lint fights); the caller's
47+
binding is conventionally **`gl`**; methods keep the **verbatim GL short name**.
48+
Call site reads `gl.DrawArrays(GL_TRIANGLES, 0, 3)` — namespace-prefix feel,
49+
original names, no rename transform.
50+
- A lowercase `struct gl` was rejected: it trips `non_camel_case_types`. The
51+
`Gl`-type + `gl`-binding pair gives the same reading with zero lint friction.
52+
- `.` (method on an owned value) is the default, not `::` (which implies a
53+
global / free function — see [dispatch modes](#dispatch-modes-mirror-c-mx--mx-global)).
54+
55+
### Constants & enums — newtype with free typed consts
56+
57+
- `#[repr(transparent)] pub struct GLenum(pub u32)` and a companion
58+
`GLbitfield` newtype with `BitOr`/`BitAnd` (for `GL_*_BIT` combining).
59+
`repr(transparent)` ⇒ ABI-identical to `u32`, zero runtime cost.
60+
- Constants are emitted as **free consts typed as the newtype**, glob-importable:
61+
```rust
62+
pub const GL_TRIANGLES: GLenum = GLenum(0x0004);
63+
// use gloam_gl::consts::*; → gl.DrawArrays(GL_TRIANGLES, 0, 3) (bare name)
64+
```
65+
This gives bare `GL_*` names (no `GLenum::` prefix) **and** call-site type
66+
safety (a raw integer won't coerce to `GLenum`). Associated consts
67+
(`GLenum::TRIANGLES`) were the alternative; identical safety, but the free-const
68+
form matches the "namespace prefix, not a rename" preference and the C output.
69+
- **True per-group enums are punted indefinitely** — GL's `group=` metadata is
70+
too incomplete to be worth the untangling, and driver-returned values would
71+
risk invalid-discriminant UB.
72+
- **Constant-typing caveat (the real cost of newtypes):** each const must be
73+
assigned a newtype, but some GL values are polymorphic — `GL_ZERO`, `GL_ONE`,
74+
`GL_NONE` are used as both enum and integer. Decision: **default those to
75+
`GLenum`** and revisit if real API usage forces casts. This is a coarse
76+
enum/bitfield/int bucketing, far smaller than full per-group typing.
77+
78+
### Safety — thin raw crate now, optional safe crate later
79+
80+
- gloam generates the **low-level crate**: raw context, PFN table, `unsafe`
81+
dispatch methods, loader, enums, extension detection. Mechanical from
82+
`FeatureSet`, in scope. Ideally `#![no_std]` (core + the loader callback).
83+
- A **safe wrapper is a separate, hand-written, downstream crate** (the `-sys` +
84+
safe convention), at the user's discretion. Safe abstractions (object
85+
ownership, slice-vs-ptr+len, `glGetError``Result`, RAII) need human judgment
86+
that can't be reliably derived from XML. Generating mechanical safety is a
87+
possible *future* project, deliberately out of the initial scope.
88+
89+
## Context representation (perf-critical)
90+
91+
The rules here are what make Rust dispatch match C's `global + offset → PFN`:
92+
93+
1. **PFN table stored inline** as a fixed `[Pfn; K]` (`K` known at generation
94+
time). **Never `Box<[…]>` / `Vec<…>`.** Inline ⇒ `self.pfns[IDX]` is one load
95+
`[base + const_offset]`. A boxed/vec table adds a second load (pointer in the
96+
struct → then the PFN) — the double-indirection to avoid.
97+
2. **`Gl` is `!Clone` and `!Copy`.** The inline table is tens of KB; deriving
98+
`Clone` would make `gl.clone()` a silent ~20 KB `memcpy`. Not deriving it
99+
forces sharing by `&Gl` / `Arc` / `static` — no accidental big copies.
100+
3. **`Arc<Gl>` adds no dispatch load.** `Arc → &data` is constant-offset pointer
101+
arithmetic, not a memory load; the inline PFN is then one load. Arc's only
102+
cost (atomic refcount) is on clone/drop, never on the hot path. (Arc is
103+
likely unnecessary, but it wouldn't harm dispatch.)
104+
105+
Pointers are 8 bytes each (like C's `pfnArray`), not GLAD's 16-byte
106+
`FnPtr{ptr,bool}`.
107+
108+
## Dispatch modes (mirror C `--mx` / `--mx-global`)
109+
110+
- **`--mx` (explicit context):** owned `Gl`, `gl.DrawArrays(...)` methods, share
111+
`&Gl`. Single indirection via the receiver register.
112+
- **`--mx-global` (primary in gloam's philosophy), as implemented:**
113+
```rust
114+
static GLOBAL: GlobalCell = ...; // UnsafeCell<Gl>, statically zero-filled
115+
unsafe { gl::load_gl_global(loader) }?; // or init_global(gl) with an owned Gl
116+
gl::DrawArrays(GL_TRIANGLES, 0, 3); // free fn over the global — the `::` form
117+
```
118+
Dispatch is `[fixed_base + offset] → PFN` — a single indirection from a
119+
link-time-fixed base, exactly like C, with **no branch and no atomic** on
120+
the hot path.
121+
- **Why not `OnceLock`** (the original sketch): the crate went `#![no_std]`
122+
(`OnceLock` is std-only), and its `get()` adds an is-initialized check C
123+
doesn't have. Instead the global is a **statically zero-filled `Gl`** in an
124+
`UnsafeCell`: reading flags before init is defined (everything reads
125+
absent), free presence queries stay *safe*, and only `init_global` is
126+
unsafe — its contract ("complete before, and never concurrent with, any
127+
other global access; publish via an ordinary happens-before edge") is the
128+
same discipline C's global already requires. Dispatching before init trips
129+
the dispatch `debug_assert` in debug builds.
130+
131+
## Matching C's size / load-time
132+
133+
Every mechanism behind gloam-C's small/fast loader has a zero-cost Rust form:
134+
135+
| Mechanism | Rust form |
136+
|---|---|
137+
| API selection (only requested APIs) | already done upstream in `resolve/` |
138+
| 8-byte PFN slots, bulk **range loading** | inline `[Pfn; K]`, same `(start,count)` range tables |
139+
| Packed **function-name blob** + offsets | `static FN_NAMES: &[u8]` + `[u16; K]` — same `.rodata` |
140+
| **Extension hashing** (XXH3-64, sort, bsearch) | `static EXT_HASHES: [u64; M]` + small (vendored/no_std) xxh3 + `binary_search` — beats GLAD's string compares |
141+
| Dispatch wrappers | `#[inline]` methods melt into call sites |
142+
| Loader | take `&mut dyn FnMut(&CStr) -> *const c_void` (one instantiation, avoids monomorphization bloat) |
143+
| Release footprint | same knobs: `panic="abort"`, LTO, strip |
144+
145+
`#[repr(C)] union` field access, if we ever want a typed named view over the
146+
same memory, is a **pure reinterpret — no runtime type/bounds check** (the only
147+
cost is the compile-time `unsafe`). So the C union trick is available at zero
148+
cost, though the inline array + const-indexed `get_unchecked` methods likely
149+
suffice.
150+
151+
## Architecture: where it plugs in
152+
153+
The resolve pipeline is backend-neutral; a Rust backend reuses all of it. See
154+
the survey in the module map ([CONTRIBUTING.md](../CONTRIBUTING.md)).
155+
156+
**Reused as-is** (no changes): selection, requirements, merge batching, command
157+
ordering optimization, PFN range tables, alias pairs, topo sort, extension
158+
hashes, indices, **structured command params** (`Param { name, type_raw }`),
159+
bootstrap-command policy.
160+
161+
**The seam** (small, additive):
162+
- `Generator::Rust(RustArgs)` variant in `src/cli.rs` (the subcommand *is* the
163+
backend selector — no `--language` flag).
164+
- `generator::rust::generate(fs, args, out, store, cmdline) -> GeneratedTree`
165+
mirroring `generator::c::generate`.
166+
- A match arm in `src/main.rs` (next to the `Generator::C` arm).
167+
- One `Generator::Rust(r) => r.alias` arm in `src/resolve/mod.rs` (the only
168+
existing coupling that reads the generator's alias flag).
169+
170+
### Type translation (the main new work)
171+
172+
The `FeatureSet` carries **C type text** in exactly three fields, which a Rust
173+
backend cannot emit directly and must translate:
174+
175+
- `Param.type_raw` (e.g. `"const GLuint *"`)
176+
- `Command.return_type` (e.g. `"const GLubyte *"`)
177+
- `TypeDef.raw_c` (whole assembled C typedefs)
178+
179+
There is **no existing type-mapping infrastructure**. So a Rust backend needs a
180+
**C-type → Rust-type translator**:
181+
182+
- **GL/GLES: small.** ~40 base types, nearly all primitive aliases —
183+
`GLuint→u32`, `GLint→i32`, `GLenum→GLenum` (newtype), `GLfloat→f32`,
184+
`GLchar→c_char`, `GLboolean→u8`, `GLsizei→i32`, `GLintptr`/`GLsizeiptr→isize`,
185+
pointers→`*const`/`*mut`, `GLDEBUGPROC`→fn pointer. Calling convention is
186+
`extern "system"` (matches `APIENTRY`). Array params (`float v[4]`) are
187+
smuggled inside `type_raw` with the name embedded — must be special-cased.
188+
- **Vulkan: large.** `TypeDef.raw_c` holds full struct/union/handle/fn-pointer
189+
bodies; re-deriving those as Rust needs either a C-text parser or **richer IR
190+
exposure** (the structured struct members are flattened into `raw_c` before
191+
`FeatureSet`). This is the dominant reason Vulkan is deferred.
192+
193+
The Rust backend also re-implements the C-policy bits from
194+
`generator/c/model.rs` (PFN-type names, parameter-string formatting) in Rust
195+
form, and its own emit layer (templates or direct codegen).
196+
197+
## Results (what was proven)
198+
199+
The original skepticism was size/load-time parity with C. Outcome, measured on
200+
the merged `gl:core=3.3,gles2=3.0` loader (2940 commands, 992 extensions),
201+
compiled to an object:
202+
203+
| section | C `gl.o` | Rust loader | note |
204+
|---|---|---|---|
205+
| `.text` | 5,304 | ~850 | dispatch is `#[inline]`, so it lives at call sites (like C's macros), not in the loader object; Rust also skips the runtime Shellsort (its known table is pre-sorted) |
206+
| `.data`/`.rodata` | 93,570 | ~92,200 | packed name blob + XXH3 table; near-identical once both carry the ext-hash table |
207+
| `.bss` | 24,552 | 0 | owned-context mode; an `OnceLock` global would add ~24 KB, matching C |
208+
209+
So: **at parity, no structural bloat.** Every early regression (fat-pointer
210+
name table; tuple-padded hash table) traced to a representation choice that was
211+
fixed back to ≤ C. The full 2940-method surface added *nothing* to the compiled
212+
object (inline methods materialize only when called).
213+
214+
Other items, all settled:
215+
216+
- **The C→Rust type translator holds across the entire GL/GLES surface**
217+
(`cargo check` clean over all 2940 signatures, incl. opaque handles, CL
218+
structs, all five callback typedefs).
219+
- **Extension detection matches C** — XXH3 reused from `Extension.hash`, hashed
220+
at runtime with `xxhash_rust::xxh3::xxh3_64`; a mock-driver test and the live
221+
hardware run both confirm it.
222+
- **Constant typing** — polymorphic `GL_ZERO`/`GL_ONE`/`GL_NONE` default to
223+
`GLenum`; no friction observed in the example.
224+
- **Real rendering**[examples/rust/gl-triangle](../examples/rust/gl-triangle/)
225+
draws a triangle via winit + glutin on a desktop GL 3.3 context (verified on
226+
an NVIDIA RTX 5090), plus a headless `--ci` pixel check.
227+
228+
## Deliberate omissions
229+
230+
- **No `--loader` (dlopen/LoadLibrary) layer.** In C that layer papers over a
231+
real portability gap; in Rust the established idiom is the downstream crate
232+
choosing [`libloading`](https://crates.io/crates/libloading) (or its
233+
windowing stack's `get_proc_address`, as the example does). Emitting one
234+
would also drag platform `dlopen` bindings into an otherwise `#![no_std]`
235+
crate. Intentionally omitted, not deferred.
236+
237+
## Open decisions
238+
239+
- **Emitter style: direct string emission vs templates.** The C backend
240+
renders minijinja templates; the Rust backend emits via `format!`/`push_str`
241+
with escaped literals. The string form is compiler-adjacent (no template
242+
runtime, rustc catches malformed interpolations) but the `\x20`-indented
243+
literals are harder to scan and review than `.j2` files. Fine at the
244+
current ~1100 lines; **revisit before Vulkan**, which would multiply the
245+
emit surface (structs, unions, handles, PFN typedefs).
246+
247+
## Deferred
248+
249+
- **Vulkan** — the `TypeDef.raw_c` struct/handle bodies are the hard part.

0 commit comments

Comments
 (0)