Skip to content

[compat] Execute simple custom generic tests through DuckDB #212

Description

@sabino

Objective

Move the already compiled custom generic-test subset into the first DuckDB execution slice for dxt test and dxt build.

Current Reality

Closed compile slices support simple root-project and installed-package column-level custom generic tests for models, sources, and seeds, using static SQL bodies with {{ model }} and {{ column_name }}. Current dxt test and dxt build intentionally reject custom generic tests before runtime execution. The compatibility matrix still lists runtime execution for custom generic tests as a planned gap.

Role and Labels

  • role: role:worker
  • labels: type:compat, area:compiler-jinja, area:macros, area:runner, area:duckdb, area:artifacts, artifact:run-results, command:test, command:build
  • readiness: status:ready, ready-for-agent

Priority

priority:p1: custom generic tests are common dbt Core project behavior, and dxt already owns the compile-time subset needed for a narrow execution path.

Readiness

Ready for one worker after the supervisor checks active runner/duckdb worktree overlap. The slice should execute only the custom tests whose compiled SQL is already supported; unsupported custom tests must continue to fail clearly.

Risk

  • risk:runtime-boundary: execution, failure counting, and run-results emission must stay in Zig.
  • risk:artifact-contract: custom-test result rows must remain Run Results v6 shaped.
  • risk:upstream-ambiguity: dbt Core custom generic-test runtime semantics should be compared for the claimed simple subset.

Dependencies / Sequencing

Build on the closed custom generic compile slices for model, source, and seed column tests. Do not launch concurrently with another active branch that owns the same DuckDB data-test execution or run-results writer files unless the supervisor records a non-overlap decision.

Acceptance Criteria

  • dxt test executes supported simple custom generic tests against existing DuckDB relations using the compiled custom-test SQL.
  • dxt build executes the same supported custom tests after their selected model/seed/source dependencies are ready in the existing supported DAG subset.
  • Passing, failing, warning, and execution-error outcomes emit deterministic Run Results v6-shaped rows for supported custom generic tests.
  • Existing built-in generic-test and singular-test execution behavior is unchanged.
  • Unsupported custom test bodies, table-level custom tests, adapter-dispatched overrides, and broad Jinja still fail clearly instead of partially executing.

Validation

  • native Zig tests for custom generic-test execution planning, failure-count wrapping, threshold classification, and run-result shape
  • focused pytest through the Zig binary for model/source/seed custom generic tests with pass/fail/warn cases
  • dbt Core oracle comparison for the simple custom-test fixture where local dbt dependencies are available
  • zig build test
  • python scripts/check_runtime_boundary.py
  • python scripts/check_public_safety.py

Stop Condition

Stop and split if execution requires a general Jinja engine, adapter dispatch execution, table-level/non-column custom tests, store_failures_as, fail_calc, SQL headers, custom materializations, bundled dbt internal macros, or Python product-runtime behavior.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions