Execute parsed unit tests in DuckDB (#217)

Author: sabino

CHANGELOG.md

@@ -70,6 +70,11 @@ compatibility.
 - Public Jaffle fixture CI now has explicit `dxt ls` and `dxt compile` gates
   alongside parse, DuckDB build/run, and docs-generate, with compile artifact
   checks for generated SQL files and dbt-shaped Manifest fields.
+- `dxt test` and test-only `dxt build` now execute selected supported unit
+  tests through DuckDB for the parsed dict row-fixture subset, materializing
+  literal `ref`/local default-quoted `source` fixtures inside a rollback-only
+  transaction, comparing projected actual rows to expected rows, and writing
+  Run Results v6 pass/fail rows.
 
 ### Fixed
 

PLAN.md

@@ -98,7 +98,7 @@ embedded DuckDB or another native adapter boundary, not Python runtime calls.
 | Runtime | Zig product runtime | Broader native adapter ABI and runner |
 | Adapter | DuckDB through a Zig-owned external CLI backend | Embedded DuckDB, Postgres, cloud adapters, cross-database planner |
 | Artifacts | `manifest.json`, `run_results.json`, `catalog.json`, `sources.json` slices | fuller dbt schemas, `semantic_manifest.json`, parse cache/state artifacts |
-| dbt resources | models, analyses in parse/list/compile, seeds, sources with schema/freshness/identifier slices, exposures, docs, macros, supported generic tests in compile and the DuckDB subset, singular SQL tests in compile and the DuckDB subset, read-only unit-test manifest/list artifacts | snapshots, fuller analysis configs/tests, full singular-test configs/patches, unit-test execution, semantic models, metrics, saved queries |
+| dbt resources | models, analyses in parse/list/compile, seeds, sources with schema/freshness/identifier slices, exposures, docs, macros, supported generic tests in compile and the DuckDB subset, singular SQL tests in compile and the DuckDB subset, unit-test manifest/list artifacts, and the first DuckDB dict-fixture unit-test execution subset | snapshots, fuller analysis configs/tests, full singular-test configs/patches, broader unit-test fixtures/overrides, semantic models, metrics, saved queries |
 | Jinja | literal, narrow scalar var-backed, and static loop-var `ref`/`source`, `doc`, inline `config`, static list `set` + simple `for`, narrow static `if`/`elif` with simple supported comparisons, selected `target`/`this` context | full parse/runtime context, macro execution, dispatch, filters, database-backed `execute`, adapter introspection |
 | Selectors and listing | names/FQN, tags, paths/files, packages, resource types, sources, exposures, unit tests, config materialization, `*`/`?`/bracket-class wildcards, `+` and `@` graph expansion, excludes, `ls` `json`/`name`/`path`/`selector` output formats, and narrow compact-JSON resource/config/identity/dependency `--output-keys` including `alias`, `identifier`, `tags`, `config.materialized`, `config.tags`, `config.enabled`, `config.docs.show`, `depends_on.nodes`, and `depends_on.macros` | YAML selectors, state/defer/result/source-status selectors, full dbt JSON and arbitrary nested `--output-keys` |
 

README.md

@@ -13,8 +13,8 @@ behavior, with DuckDB as the first deterministic execution adapter.
 | `dxt compile` | Partial | Compiles selected enabled SQL models, selected analyses, selected supported built-in generic tests, root-project model/source/seed column custom generic tests, installed-package model/source/seed column custom generic tests selected through `package.test_name`, and selected singular SQL tests through the supported render-only Jinja subset and writes compiled SQL plus manifest fields without opening DuckDB. | Non-column and table-level custom generic tests, full Jinja, general macro execution, adapter dispatch execution, arbitrary expressions, filters, hooks. |
 | `dxt run` | Partial | Executes selected enabled DuckDB SQL models with `table` and `view` materializations; writes completed prior rows plus a sanitized `error` run-result row when a selected model fails during DuckDB execution, records `skipped` rows for selected blocked model descendants, and continues later selected independent models. | Seeds, tests, snapshots, incremental, ephemeral, hooks, grants, full materialization macros. |
 | `dxt seed` | Partial | Executes selected root-project and installed-package DuckDB CSV seeds, honors supported seed YAML `quote_columns` and `column_types` configs for DuckDB CSV loading, writes `manifest.json` and seed-shaped `run_results.json`, filters mixed selections to seed resources, and rejects selections that match no seeds before opening DuckDB. | Hooks, grants, full-refresh semantics, full materialization macros, broader seed config parity. |
-| `dxt test` | Partial | Executes selected supported DuckDB generic tests and singular SQL tests against existing target relations and writes `manifest.json` plus `run_results.json`; supported built-in model/seed/source generic tests and singular SQL tests honor `where`, `limit`, `severity`, `warn_if`, `error_if`, and `store_failures` for simple failure-count thresholds and deterministic DuckDB audit-table creation/drop. | Unit-test execution, custom generic macros, `store_failures_as`, building parents before tests, broader singular config parity. |
-| `dxt build` | Partial | Executes root-project and installed-package CSV seeds with supported seed YAML `quote_columns` and `column_types` configs, selected DuckDB models, supported model/source/seed column generic tests, singular SQL tests, explicit table-level `column_name` built-in tests, source-target `relationships`, and mixed selected seed/model/test subsets; supported built-in model/seed/source generic tests and singular SQL tests honor `where`, `limit`, `severity`, `warn_if`, `error_if`, and `store_failures`; writes sanitized model/seed execution-error rows and `skipped` rows for selected blocked descendants/tests while continuing selected independent supported seeds, models, and data tests after execution or data-test failures. | Full dbt queue semantics, broader seed config parity, wider tests, broader singular configs, unit-test execution, generic-test runtime errors, full indirect-selection modes, `store_failures_as`. |
+| `dxt test` | Partial | Executes selected supported DuckDB generic tests and singular SQL tests against existing target relations plus selected supported dict-fixture unit tests in rollback-only DuckDB transactions, then writes `manifest.json` plus `run_results.json`; supported built-in model/seed/source generic tests and singular SQL tests honor `where`, `limit`, `severity`, `warn_if`, `error_if`, and `store_failures` for simple failure-count thresholds and deterministic DuckDB audit-table creation/drop. | CSV/SQL unit-test fixtures, unit-test overrides, custom generic macros, `store_failures_as`, building parents before data tests, broader singular config parity. |
+| `dxt build` | Partial | Executes root-project and installed-package CSV seeds with supported seed YAML `quote_columns` and `column_types` configs, selected DuckDB models, supported model/source/seed column generic tests, singular SQL tests, explicit table-level `column_name` built-in tests, source-target `relationships`, test-only selected supported unit tests, and mixed selected seed/model/test subsets; supported built-in model/seed/source generic tests and singular SQL tests honor `where`, `limit`, `severity`, `warn_if`, `error_if`, and `store_failures`; writes sanitized model/seed execution-error rows and `skipped` rows for selected blocked descendants/tests while continuing selected independent supported seeds, models, and data tests after execution or data-test failures. | Full dbt queue semantics, broader seed config parity, wider tests, broader singular configs, CSV/SQL unit-test fixtures, mixed model/seed plus unit-test scheduling, generic-test runtime errors, full indirect-selection modes, `store_failures_as`. |
 | `dxt docs generate` | Partial | Writes `manifest.json`, compiled SQL, and `catalog.json`; introspects selected existing DuckDB model/seed/source relations when available, including resolved source relation identity from supported project-level and YAML source/table configs in the current DuckDB catalog contract. | Docs-time execution, comments/owners/stats, richer source metadata beyond relation identity, bundled dbt docs UI assets. |
 | `dxt docs serve` | Partial | Serves generated target-directory docs artifacts over localhost HTTP with `--host`, `--port`, `--no-browser`, `--browser`, and `--no-open` parsing; writes a small dxt-owned `index.html`; does not mutate `manifest.json` or `catalog.json`. | Browser opening, dbt's bundled docs SPA, Fusion docs v2/index API server, live reload, richer static asset handling. |
 | `dxt source freshness` | Partial | Queries selected DuckDB source tables with resolved source relation identity, `loaded_at_field`, `loaded_at_query`, and freshness settings from supported project-level and YAML source/table configs, then writes Sources v3-shaped results. Selected DuckDB sources with freshness thresholds but no loaded-at config write dbt-shaped runtime-error rows because DuckDB metadata-based freshness is unsupported. Its selector preflight can reuse `source_status:pass` / `source_status:warn` / `source_status:error` when `--state` points at a prior `sources.json`. | Metadata freshness execution, Jinja in freshness queries beyond narrow source schema rendering, concurrency, non-DuckDB adapters. |
@@ -49,7 +49,7 @@ behavior, with DuckDB as the first deterministic execution adapter.
 | Docs blocks | Partial | Markdown docs block parsing and literal `doc()` descriptions. | Dynamic doc expressions and full dbt docs UI behavior. |
 | Generic tests | Partial | Manifest nodes, `compile` artifacts, and DuckDB execution for supported model, seed, and source column built-ins plus explicit table-level `column_name` built-ins, including explicit `accepted_values` `quote: false`, literal source-target `relationships`, and supported `where` / `limit` / severity threshold / `store_failures` configs. Compile-only custom model, source, and seed column tests are supported for root-project macros and installed-package macros called with `package.test_name`, when the body contains only static SQL plus `{{ model }}` and `{{ column_name }}`. | Runtime execution for custom generic tests, non-column and table-level custom generic tests, `store_failures_as`, typed scalar value artifact parity, adapter-dispatched test overrides, general threshold expression evaluation. |
 | Singular tests | Partial | SQL files under configured `test-paths` are parsed as singular data tests, excluding `generic/` and `fixtures/`; literal inline `config(enabled=false)` and supported top-level `tests:` / `data_tests:` YAML patches can disable singular tests into `manifest.disabled`; manifest nodes omit generic-only fields while carrying supported description, tag, enabled, severity, threshold, `where`, `limit`, and `store_failures` config fields; selectors support `test_type:singular`, `test_type:data`, and patched tags; `compile` writes selected compiled artifact paths; and DuckDB `build`/`test` executes them through failure-row counting and optional audit-table persistence. | Dynamic enabled expressions, broader singular config parity, `store_failures_as`, indirect-selection parity, broader Jinja/macros. |
-| Unit tests | Partial | Read-only YAML parsing for dict-style `given`/`expect` row fixtures, Manifest v12-shaped `unit_tests`, parent/child maps, `resource_type:unit_test`, `unit_test:`, and `test_type:unit` listing. | Execution, fixture materialization, CSV/SQL fixtures, overrides, version expansion, disabled-unit-test placement, SQL comparison, and run-results. |
+| Unit tests | Partial | YAML parsing for dict-style `given`/`expect` row fixtures, Manifest v12-shaped `unit_tests`, parent/child maps, `resource_type:unit_test`, `unit_test:`, and `test_type:unit` listing. `dxt test` and test-only `dxt build` execute selected supported unit tests through DuckDB by materializing literal `ref`/default-quoted local `source` dict fixtures in a rollback-only transaction, comparing projected actual rows to expected rows with `except all`, and writing Run Results v6 pass/fail rows. | CSV/SQL fixtures, overrides, model version expansion, disabled-unit-test placement changes, adapter-specific source quoting/database fixture materialization, mixed model/seed plus unit-test build scheduling, and broader dbt unit-test introspection parity. |
 | Snapshots, semantic models, metrics, saved queries, functions, groups | Planned | Not first-class yet or only empty artifact maps where needed. | Full parser, graph, artifact, and execution semantics. |
 
 ## Jinja And Macros