Add minimal macro dispatch rendering (#86)

Author: sabino

.agent/research/m2-minimal-macro-dispatch-rendering.md

@@ -0,0 +1,53 @@
+# M2 Minimal Macro Dispatch Rendering
+
+Date: 2026-06-17
+
+## Upstream Reference
+
+- dbt Core v1: runtime macro rendering and namespace behavior in
+  `core/dbt/context/providers.py`, `core/dbt/context/macros.py`, and
+  `core/dbt/clients/jinja.py`.
+- dbt Fusion direction: compile-node context construction in
+  `crates/dbt-jinja-utils/src/phases/compile/compile_node_context.rs`.
+- Public fixture shape: Fusion Jaffle Shop
+  `crates/dbt-init/assets/jaffle_shop/macros/cents_to_dollars.sql`.
+
+## dxt Scope
+
+This slice adds a narrow Zig-only compile/runtime path for Jaffle-style macro
+dispatch:
+
+- model expression calls such as `{{ cents_to_dollars('subtotal') }}`;
+- wrapper macro bodies shaped as
+  `{{ return(adapter.dispatch('cents_to_dollars')(column_name)) }}`;
+- selected adapter/default implementation macro bodies that are static SQL
+  templates with positional parameter interpolation such as
+  `{{ column_name }}`.
+
+Dispatch selection reuses the existing Zig resolver and adapter-prefix rules.
+DuckDB falls back to `default__*` unless a `duckdb__*` implementation exists.
+
+## Explicit Boundaries
+
+This is not a general Jinja or dbt macro interpreter. The compiler still
+rejects statement tags inside macro bodies, unsupported dynamic arguments,
+general conditionals, loops, filters, runtime context objects, materialization
+execution, arbitrary return values, and non-dispatch nested macro execution.
+
+Python remains test/oracle tooling only; all product behavior is in Zig.
+
+## Validation
+
+- Native Zig tests cover wrapper dispatch rendering, adapter-specific dispatch
+  preference, return-wrapper dependency scanning, and unsupported macro
+  statements.
+- Python integration coverage uses a synthetic Jaffle-style fixture through
+  `compile`, `docs generate`, `run`, and `build`, checking compiled SQL,
+  manifest macro dependencies, run-results schema slices, and DuckDB output.
+
+## Follow-Up
+
+The next macro slices should handle upstream source-grounded package namespace
+edge cases, macro defaults/keyword arguments, and broader parse-vs-runtime
+Jinja boundaries only as separate PRs with their own dbt Core/Fusion reference
+notes and stop conditions.

CHANGELOG.md

@@ -15,6 +15,10 @@ compatibility.
   checksums.
 - Source-grounded upstream reference-map refresh with the next five small
   dbt Core/Fusion-backed compatibility slices.
+- Narrow Zig compile/runtime rendering for Jaffle-style macro dispatch wrappers
+  such as `cents_to_dollars`, including adapter/default implementation
+  selection, manifest macro dependency coverage, and DuckDB run/build
+  integration coverage.
 
 ## 0.0.0-pre-alpha
 

PLAN.md

@@ -890,7 +890,7 @@ Exit criteria:
 - CI format validation now covers every tracked Zig source file under `src/`, including extracted `src/project/*.zig` modules, so M1A module splits remain under the same formatting gate as the root CLI files.
 - `dxt parse` now targets the supported Tier 0 subset: project name/model paths/seed paths/macro paths, target path, narrow top-level scalar project `vars`, CLI `--vars` scalar overrides, project and package model path configs for literal `+materialized`, `+tags`, and model/seed `+docs.node_color`, root-project model config overrides for installed packages, SQL model discovery, CSV seed discovery, installed package SQL model and CSV seed discovery from `dbt_packages`, source discovery, installed package source discovery, exposure discovery, installed package exposure discovery, project macro discovery, dbt-shaped generic test macro and materialization macro block discovery, installed package macro discovery from `dbt_packages`, macro property YAML for project macro descriptions, arguments, `docs`, and scalar JSON-compatible `meta`, project and package docs block discovery, literal and narrow scalar `var('name')` / `var('name', 'default')`-backed `ref` to models or seeds, two-argument package refs, package-local refs in installed package models and exposures, unique installed-package fallback for unqualified refs, literal and narrow scalar `var('name')` / `var('name', 'default')`-backed `source`, package-local sources in installed package models, unique installed-package fallback for unqualified sources, literal `doc` in project and package descriptions, inline `config(materialized=..., tags=...)`, known project/package-qualified/package-local macro call dependencies, narrow project and package YAML model properties for scalar descriptions, simple columns, tags, materialization, disabled SQL models, dbt-shaped `unique`, `not_null`, `accepted_values`, and `relationships` generic test nodes, model/test `refs` and `sources` artifact fields, dependency maps, and deterministic partial `manifest.json`. The manifest includes the v12 top-level maps needed by the M1 artifact shape and is covered by a pinned local dbt Manifest v12 schema slice. YAML generic test arguments are currently supported for scalar values plus inline and block lists required by public Jaffle Shop DuckDB-style tests. Full dbt `var()` semantics remain explicitly deferred until the parse/compile Jinja context work covers package scoping, `vars.yml`, non-string values, rendered var values, `has_var`, missing-var parse/runtime behavior, project/profile rendering, and partial-parse invalidation.
 - `dxt ls` now lists dbt-selectable resources from the same parser graph, including scalar project/CLI var-resolved dependency edges, with stable text/JSON output and basic name/FQN wildcards, tag wildcards, slash-aware `path:` wildcards, exact `package:`/`package:this`, `source:` wildcards including package-qualified source selectors, `exposure:` wildcards, `resource_type:`, `test_type:generic`, config materialization, comma intersection, whitespace union, multi-argument selector lists, repeated selector flags, leading/trailing `+` graph expansion, and exact exclude filters; macros are emitted in artifacts but not exposed as `ls` resources.
-- `dxt compile` has started as a render-only M2 boundary for the current graph subset. It loads and resolves the same Zig parser graph, applies `--select` and `--exclude`, compiles selected enabled SQL model nodes, writes compiled SQL under `target/compiled/<package>/...`, and emits `compiled`, `compiled_code`, `compiled_path`, `relation_name`, `extra_ctes`, and `extra_ctes_injected` only for compiled model nodes. The current compiler renders `config` to empty text, literal or narrow scalar var-backed `ref`/`source` calls to deterministic quoted relation names, profile-derived `target.*`, current-model `this`, quoted literal inline `config(schema=..., alias=...)` as default dbt relation schema/identifier components, and a narrow source-grounded compile-time Jinja subset for unescaped `{% set name = ['string', ...] %}` string lists plus `{% for item in name %}` body expansion without opening a database connection. This static list-loop slice is documented in `.agent/research/m2-static-jinja-set-for-loops.md` and intentionally rejects scalar set values, unquoted or escaped list entries, filters, conditionals, loop metadata, macros, dynamic lists, and general expression evaluation.
+- `dxt compile` has started as a render-only M2 boundary for the current graph subset. It loads and resolves the same Zig parser graph, applies `--select` and `--exclude`, compiles selected enabled SQL model nodes, writes compiled SQL under `target/compiled/<package>/...`, and emits `compiled`, `compiled_code`, `compiled_path`, `relation_name`, `extra_ctes`, and `extra_ctes_injected` only for compiled model nodes. The current compiler renders `config` to empty text, literal or narrow scalar var-backed `ref`/`source` calls to deterministic quoted relation names, profile-derived `target.*`, current-model `this`, quoted literal inline `config(schema=..., alias=...)` as default dbt relation schema/identifier components, and a narrow source-grounded compile-time Jinja subset for unescaped `{% set name = ['string', ...] %}` string lists plus `{% for item in name %}` body expansion without opening a database connection. It also renders a narrow Jaffle-style macro dispatch subset for literal model macro calls such as `{{ cents_to_dollars('subtotal') }}`, wrapper bodies shaped as `return(adapter.dispatch(...)(column_name))`, and selected adapter/default implementation bodies with positional parameter interpolation. The static list-loop slice is documented in `.agent/research/m2-static-jinja-set-for-loops.md`; the minimal macro dispatch slice is documented in `.agent/research/m2-minimal-macro-dispatch-rendering.md`. The compiler intentionally rejects scalar set values, unquoted or escaped list entries, filters, conditionals, loop metadata, dynamic lists, general expression evaluation, statement tags inside macro bodies, materialization macro execution, and arbitrary macro runtime behavior.
 - `dxt docs generate` has started as a docs artifact boundary. It loads and resolves the same Zig parser graph, applies `--select` and `--exclude` to compiled model output, writes compiled SQL, writes `manifest.json`, and writes dbt-shaped `catalog.json`. The catalog remains empty when no local DuckDB database exists or when selected relations are absent, and includes selected model/seed node relation metadata plus selected source relation metadata and ordered columns when an existing target DuckDB file can be introspected through the Zig-owned DuckDB CLI backend. Macro execution, docs-time materialization, tests, source relation config, source freshness inside docs, comments, owners, richer stats, non-DuckDB adapters, `run_results.json`, and `docs serve` remain out of scope.
 - `dxt source freshness` has started the M3 DuckDB `sources.json` execution path. It loads and resolves the same Zig parser graph, applies supported source selectors/excludes, filters to source nodes with table-level freshness criteria, queries selected DuckDB source tables through table-level `loaded_at_field` SQL text plus optional raw `freshness.filter` SQL or through table-level raw `loaded_at_query` SQL, classifies `pass` / `warn` / `error` from `warn_after` and `error_after`, writes `manifest.json`, writes dbt-shaped `sources.json` v3 success rows including stale empty/all-null loaded-at results, writes dbt-shaped runtime-error rows for unsupported per-source execution gaps such as missing loaded-at configuration or conflicting `loaded_at_field`/`loaded_at_query`, and returns exit code `1` when any freshness status is `error` or runtime error. Source-level inheritance, Jinja rendering inside `loaded_at_query`, metadata freshness, `config:` overrides, source-status selectors, hooks, threaded scheduling, non-DuckDB adapters, and embedded `libduckdb` remain future source-grounded slices.
 - `dxt run` has started the M3 DuckDB execution path for selected enabled SQL models. It loads and resolves the same Zig parser graph, applies supported selectors/excludes, compiles selected SQL models, validates that selected models use only `table` or `view` materializations before opening DuckDB, executes selected models in dependency order through a Zig-owned external DuckDB CLI backend, writes compiled SQL, writes `manifest.json`, and writes a minimal dbt-shaped success-only `run_results.json` v6 slice after completed runs. It supports default `target/dxt.duckdb` output plus scalar DuckDB profile `path` resolved relative to the loaded `profiles.yml` directory as a deterministic dxt-local path-base choice for this first CLI-backed slice. It does not execute seeds, tests, snapshots, incremental, ephemeral, hooks, grants, docs persistence, catalog introspection, failure/partial run-results artifacts, relation staging/backup rename parity, threaded scheduling, `:memory:`, MotherDuck, or embedded `libduckdb`.
@@ -900,4 +900,4 @@ Exit criteria:
 - Selector wildcard behavior is currently pinned to observed dbt Core 1.10 behavior. dbt Fusion preview currently differs for resource-type-prefixed wildcard selectors such as `model.<package>.*` and filename-suffix path selectors such as `path:*orders.sql`; a future Fusion-compatibility slice must decide whether to support a selector dialect switch or a compatible superset.
 - Compatibility planning now uses a source-grounded reference map under `.agent/research/dbt-upstream-reference-map.md`; future feature slices should name upstream dbt v1/v2 source references, dxt Zig owners, affected artifact fields, validation gates, and stop conditions before implementation.
 - The committed dbt Core M1 oracle harness lives in `scripts/check_dbt_core_m1_oracle.py`. It is optional developer-side Python tooling that requires `dbt-core` and `dbt-duckdb`, invokes dbt Core through its Python runner, runs `dxt parse` through the Zig binary, and compares stable manifest slices for the supported synthetic M1 fixture ladder. It ignores dbt internal package docs/macros that are outside the current dxt artifact scope, records a known allowed gap for installed-package exposure refs that dbt Core resolves to a root same-name model while dxt currently resolves package-local, and leaves full source-map parity, full artifact schemas, and execution parity for later slices.
-- Before broadening M2 product implementation, close or explicitly re-scope the remaining M1 macro-compatibility behavior gaps. Macro `docs`/`meta` patch fields are covered for the current scalar artifact subset. Macro argument extraction under dbt Core v1 `flags.validate_macro_args` semantics and YAML patch argument validation/replacement are implemented for the manifest artifact surface. Static macro dependency lookup now uses the supported dbt order of current package, root project, other-package fallback for macro bodies, graph-present internal `dbt` macros, and literal `adapter.dispatch(...)` dependency extraction. Parse-time dispatch prefixes now come from a narrow source-grounded `profiles.yml` adapter identity parser and emit manifest `metadata.adapter_type`, with default DuckDB behavior preserved when no profile file is loaded. Root-project `dispatch:` config search order is now honored for static `adapter.dispatch(...)` dependency extraction. Macro execution, bundled dbt internal macros, full target context, credential validation, and materialization runtime lookup remain planned. `{% data_test %}` has native source-grounded parser coverage, but the local dbt Core 1.10 oracle rejects that tag before writing artifacts, so dbt-oracle coverage currently pins `{% test %}` and `{% materialization %}` block parity.
+- Before broadening M2 product implementation, close or explicitly re-scope the remaining M1 macro-compatibility behavior gaps. Macro `docs`/`meta` patch fields are covered for the current scalar artifact subset. Macro argument extraction under dbt Core v1 `flags.validate_macro_args` semantics and YAML patch argument validation/replacement are implemented for the manifest artifact surface. Static macro dependency lookup now uses the supported dbt order of current package, root project, other-package fallback for macro bodies, graph-present internal `dbt` macros, literal `adapter.dispatch(...)` dependency extraction, and return-wrapper dispatch dependency extraction. Parse-time dispatch prefixes now come from a narrow source-grounded `profiles.yml` adapter identity parser and emit manifest `metadata.adapter_type`, with default DuckDB behavior preserved when no profile file is loaded. Root-project `dispatch:` config search order is now honored for static `adapter.dispatch(...)` dependency extraction. Compile/runtime macro rendering is limited to the Jaffle-style dispatch wrapper subset documented in `.agent/research/m2-minimal-macro-dispatch-rendering.md`; bundled dbt internal macros, full target context inside macros, credential validation, general macro execution, and materialization runtime lookup remain planned. `{% data_test %}` has native source-grounded parser coverage, but the local dbt Core 1.10 oracle rejects that tag before writing artifacts, so dbt-oracle coverage currently pins `{% test %}` and `{% materialization %}` block parity.