Handle unsupported DuckDB metadata freshness

CHANGELOG.md

@@ -69,6 +69,10 @@ compatibility.
 
 ### Fixed
 
+- DuckDB `source freshness` now reports selected sources that have freshness
+  thresholds but no `loaded_at_field` or `loaded_at_query` as dbt-shaped
+  Sources v3 runtime-error rows explaining that DuckDB does not support
+  metadata-based freshness, while continuing other selected loaded-at checks.
 - First DuckDB-compatible `ephemeral` model slice: downstream SQL models now
   compile supported ephemeral parents into deterministic injected CTEs, emit
   Manifest `extra_ctes` / `extra_ctes_injected`, and run/build without creating

PLAN.md

@@ -1498,7 +1498,7 @@ Exit criteria:
   materialization macros, adapter-specific relation staging, incremental,
   snapshots, and broader scheduler semantics.
 - `dxt docs generate` has started as a docs artifact boundary. It loads and resolves the same Zig parser graph, applies `--select` and `--exclude` to compiled model output, writes compiled SQL, writes `manifest.json`, and writes dbt-shaped `catalog.json`. The catalog remains empty when no local DuckDB database exists or when selected relations are absent, and includes selected model/seed node relation metadata plus selected source relation metadata and ordered columns when an existing target DuckDB file can be introspected through the Zig-owned DuckDB CLI backend, including configured source database matching through DuckDB `table_catalog` for the current source relation identity contract and supported project-level/YAML source config inheritance. `dxt docs serve` has started as a static target-directory HTTP server for existing generated artifacts. Macro execution, docs-time materialization, tests, source freshness inside docs, comments, owners, richer stats, non-DuckDB adapters, `run_results.json`, dbt's bundled docs SPA, browser opening, and Fusion docs v2 endpoints remain out of scope.
-- `dxt source freshness` has started the M3 DuckDB `sources.json` execution path. It loads and resolves the same Zig parser graph, applies supported source selectors/excludes including `source_status:pass` / `source_status:warn` / `source_status:error` from a prior dbt Sources v3 `--state/sources.json`, filters to source nodes with resolved source/table freshness criteria, queries selected DuckDB source tables through resolved relation identity plus resolved `loaded_at_field` SQL text and optional raw `freshness.filter` SQL or through resolved raw `loaded_at_query` SQL, classifies `pass` / `warn` / `error` from `warn_after` and `error_after`, writes `manifest.json`, writes dbt-shaped `sources.json` v3 success rows including stale empty/all-null loaded-at results, writes dbt-shaped runtime-error rows for unsupported per-source execution gaps such as missing loaded-at configuration, and returns exit code `1` when any freshness status is `error` or runtime error. Root-project `dbt_project.yml` `sources:` configs for supported relation/freshness fields, source/table `config:` inheritance for `loaded_at_field`, `loaded_at_query`, and `freshness`, dbt-shaped threshold inheritance, final `freshness: null`, narrow `schema: "{{ target.schema }}"` rendering, source table `identifier` physical-name overrides, source/table database and database/schema/identifier quoting relation identity, resolved source database/schema/identifier use in compile/catalog/freshness/test paths, and expanded source manifest fields are implemented. General Jinja inside `loaded_at_query`, metadata freshness, hooks, threaded scheduling, installed-package project source config application, non-DuckDB adapters, `state:`, `result:`, defer, and embedded `libduckdb` remain future source-grounded slices.
+- `dxt source freshness` has started the M3 DuckDB `sources.json` execution path. It loads and resolves the same Zig parser graph, applies supported source selectors/excludes including `source_status:pass` / `source_status:warn` / `source_status:error` from a prior dbt Sources v3 `--state/sources.json`, filters to source nodes with resolved source/table freshness criteria, queries selected DuckDB source tables through resolved relation identity plus resolved `loaded_at_field` SQL text and optional raw `freshness.filter` SQL or through resolved raw `loaded_at_query` SQL, classifies `pass` / `warn` / `error` from `warn_after` and `error_after`, writes `manifest.json`, writes dbt-shaped `sources.json` v3 success rows including stale empty/all-null loaded-at results, writes dbt-shaped runtime-error rows for unsupported per-source execution gaps including DuckDB sources with freshness thresholds but no `loaded_at_field` or `loaded_at_query`, and returns exit code `1` when any freshness status is `error` or runtime error. Root-project `dbt_project.yml` `sources:` configs for supported relation/freshness fields, source/table `config:` inheritance for `loaded_at_field`, `loaded_at_query`, and `freshness`, dbt-shaped threshold inheritance, final `freshness: null`, narrow `schema: "{{ target.schema }}"` rendering, source table `identifier` physical-name overrides, source/table database and database/schema/identifier quoting relation identity, resolved source database/schema/identifier use in compile/catalog/freshness/test paths, and expanded source manifest fields are implemented. General Jinja inside `loaded_at_query`, metadata freshness execution, hooks, threaded scheduling, installed-package project source config application, non-DuckDB adapters, `state:`, `result:`, defer, and embedded `libduckdb` remain future source-grounded slices.
 - `dxt run` has started the M3 DuckDB execution path for selected enabled SQL models. It loads and resolves the same Zig parser graph, applies supported selectors/excludes, compiles selected SQL models, validates that selected models use only `table` or `view` materializations before opening DuckDB, executes selected models in dependency order through a Zig-owned external DuckDB CLI backend, writes compiled SQL, writes `manifest.json`, and writes a minimal dbt-shaped `run_results.json` v6 slice after completed runs. When a selected model fails with a DuckDB execution error, it writes completed prior rows plus a sanitized `status: "error"` row for the failed model, records `status: "skipped"` rows for selected blocked model descendants that survived `--exclude`, continues executing later selected models that do not depend on the failed node, writes `run_results.json`, and returns exit code `1`. It supports default `target/dxt.duckdb` output plus scalar DuckDB profile `path` resolved relative to the loaded `profiles.yml` directory as a deterministic dxt-local path-base choice for this first CLI-backed slice. It does not execute seeds, tests, snapshots, incremental, ephemeral, hooks, grants, docs persistence, catalog introspection, seed independent-resource continuation after failure, relation staging/backup rename parity, threaded scheduling, `:memory:`, MotherDuck, or embedded `libduckdb`.
 - `dxt seed` has started the M3 DuckDB seed command path for selected root-project and installed-package CSV seeds. It loads and resolves the same Zig parser graph, applies supported selectors/excludes, filters mixed selections to seed resources in the dbt `SeedTask` / `ResourceTypeSelector` shape, rejects selections that match no seeds before opening DuckDB, writes `manifest.json`, loads selected seeds through the existing Zig-owned DuckDB CLI backend from the loaded root or package project root, writes seed-shaped Run Results v6 rows with null compiled fields, and prints a seed-specific success summary. Package seed execution supports name, package, and dependency selector paths when the final runnable set is seed resources. Supported seed YAML `quote_columns` and `column_types` configs are parsed for root-project and installed-package CSV seeds, emitted in dbt-shaped Manifest seed configs, and applied through DuckDB CSV name-normalization and type-map options. Hooks, grants, docs persistence, full-refresh semantics, full materialization macro execution, threaded scheduling, broader seed config parity, and embedded `libduckdb` remain future source-grounded slices.
 - `dxt test` has started the M3 DuckDB data-test command path. It loads and resolves the same Zig parser graph, filters selection to test resources, executes selected supported DuckDB generic tests and enabled singular SQL tests against already-existing target relations through the existing Zig-owned DuckDB CLI backend, writes `manifest.json`, writes Run Results v6-shaped pass/fail/warn rows, and returns exit code `1` when any selected test fails while warning tests do not fail the command. It supports the built-in `not_null`, `unique`, `accepted_values`, and `relationships` generic-test subset already implemented for `build`, including model/seed/source generic-test `where`, `limit`, `severity`, `warn_if`, `error_if`, and `store_failures` configs for simple failure-count threshold comparisons and deterministic DuckDB audit-table persistence, plus singular SQL test files discovered from configured `test-paths` while skipping `generic/` and `fixtures/` subdirectories. Supported singular YAML patches can set description, config tags, enabled, `where`, `limit`, severity, warn/error thresholds, and `store_failures`; `where` and `limit` are applied to failure-row SQL, and threshold classification matches the existing supported generic-test model. Literal inline or YAML `config(enabled=false)` singular SQL tests are preserved under `manifest.disabled` and omitted from active selectors and execution, with inline enabled config taking precedence over YAML patch enabled config; literal inline singular `config(store_failures=true|false)` is also supported. Singular manifest nodes intentionally omit generic-only fields such as `test_metadata`, `column_name`, and `attached_node`, and selectors now cover `test_type:singular`, `test_type:data`, and patched singular tags. It does not build parent models or seeds, execute unit tests, run custom generic-test macros, support dynamic singular `enabled`, support `store_failures_as`, broaden singular config parity, change indirect-selection semantics, or use Python product runtime behavior.

docs/COMPATIBILITY.md

@@ -17,7 +17,7 @@ behavior, with DuckDB as the first deterministic execution adapter.
 | `dxt build` | Partial | Executes root-project and installed-package CSV seeds with supported seed YAML `quote_columns` and `column_types` configs, selected DuckDB models, supported model/source/seed column generic tests, singular SQL tests, explicit table-level `column_name` built-in tests, source-target `relationships`, and mixed selected seed/model/test subsets; supported built-in model/seed/source generic tests and singular SQL tests honor `where`, `limit`, `severity`, `warn_if`, `error_if`, and `store_failures`; writes sanitized model/seed execution-error rows and `skipped` rows for selected blocked descendants/tests while continuing selected independent supported seeds, models, and data tests after execution or data-test failures. | Full dbt queue semantics, broader seed config parity, wider tests, broader singular configs, unit-test execution, generic-test runtime errors, full indirect-selection modes, `store_failures_as`. |
 | `dxt docs generate` | Partial | Writes `manifest.json`, compiled SQL, and `catalog.json`; introspects selected existing DuckDB model/seed/source relations when available, including resolved source relation identity from supported project-level and YAML source/table configs in the current DuckDB catalog contract. | Docs-time execution, comments/owners/stats, richer source metadata beyond relation identity, bundled dbt docs UI assets. |
 | `dxt docs serve` | Partial | Serves generated target-directory docs artifacts over localhost HTTP with `--host`, `--port`, `--no-browser`, `--browser`, and `--no-open` parsing; writes a small dxt-owned `index.html`; does not mutate `manifest.json` or `catalog.json`. | Browser opening, dbt's bundled docs SPA, Fusion docs v2/index API server, live reload, richer static asset handling. |
-| `dxt source freshness` | Partial | Queries selected DuckDB source tables with resolved source relation identity, `loaded_at_field`, `loaded_at_query`, and freshness settings from supported project-level and YAML source/table configs, then writes Sources v3-shaped results. Its selector preflight can reuse `source_status:pass` / `source_status:warn` / `source_status:error` when `--state` points at a prior `sources.json`. | Metadata freshness, Jinja in freshness queries beyond narrow source schema rendering, concurrency, non-DuckDB adapters. |
+| `dxt source freshness` | Partial | Queries selected DuckDB source tables with resolved source relation identity, `loaded_at_field`, `loaded_at_query`, and freshness settings from supported project-level and YAML source/table configs, then writes Sources v3-shaped results. Selected DuckDB sources with freshness thresholds but no loaded-at config write dbt-shaped runtime-error rows because DuckDB metadata-based freshness is unsupported. Its selector preflight can reuse `source_status:pass` / `source_status:warn` / `source_status:error` when `--state` points at a prior `sources.json`. | Metadata freshness execution, Jinja in freshness queries beyond narrow source schema rendering, concurrency, non-DuckDB adapters. |
 | `version`, help | Supported | Basic CLI metadata and help. | Release version stamping beyond current build metadata. |
 | `debug`, `deps`, `init`, `run-operation`, `snapshot`, `retry`, `clone` | Planned | Not implemented. | Command-specific dbt parity. |
 

src/project.zig

@@ -306,7 +306,7 @@ pub fn sourceFreshness(runtime: Runtime, options: Options, stdout: *Io.Writer, s
                 continue;
             };
             if (source.loaded_at_field == null and source.loaded_at_query == null) {
-                try appendSourceFreshnessRuntimeError(runtime.allocator, &results, source, "source freshness currently requires loaded_at_field or loaded_at_query");
+                try appendSourceFreshnessRuntimeError(runtime.allocator, &results, source, source_freshness.unsupported_metadata_freshness_message);
                 had_failure = true;
                 continue;
             }

src/project/source_freshness.zig

@@ -43,6 +43,8 @@ pub const SourceStatusIndex = struct {
     }
 };
 
+pub const unsupported_metadata_freshness_message = "source freshness requires loaded_at_field or loaded_at_query because the DuckDB adapter does not support metadata-based freshness";
+
 pub fn deinitResults(allocator: std.mem.Allocator, results: []const CheckResult) void {
     for (results) |result| {
         if (result.max_loaded_at) |value| allocator.free(value);
@@ -118,7 +120,9 @@ pub fn validateThreshold(threshold: FreshnessThreshold) !void {
 }
 
 pub fn unsupportedExecutionReason(source: *const SourceDef) ?[]const u8 {
-    _ = source;
+    if (source.freshness != null and source.loaded_at_field == null and source.loaded_at_query == null) {
+        return unsupported_metadata_freshness_message;
+    }
     return null;
 }
 
@@ -258,6 +262,38 @@ test "source freshness allows loaded_at_query to take precedence over inherited
     try std.testing.expect(unsupportedExecutionReason(&source) == null);
 }
 
+test "source freshness reports unsupported DuckDB metadata freshness reason" {
+    const source = SourceDef{
+        .package_name = "demo",
+        .unique_id = "source.demo.raw.orders",
+        .source_name = "raw",
+        .table_name = "orders",
+        .original_file_path = "models/schema.yml",
+        .freshness = .{
+            .warn_after = .{ .count = 1, .period = "hour" },
+            .error_after = .{ .count = 1, .period = "day" },
+        },
+    };
+
+    const reason = unsupportedExecutionReason(&source).?;
+    try std.testing.expectEqualStrings(unsupported_metadata_freshness_message, reason);
+
+    const rendered = try renderSources(std.testing.allocator, &.{.{
+        .source = &source,
+        .status = "runtime error",
+        .error_message = reason,
+    }});
+    defer std.testing.allocator.free(rendered);
+
+    var parsed = try std.json.parseFromSlice(std.json.Value, std.testing.allocator, rendered, .{});
+    defer parsed.deinit();
+    const result = parsed.value.object.get("results").?.array.items[0].object;
+    try std.testing.expectEqualStrings("source.demo.raw.orders", result.get("unique_id").?.string);
+    try std.testing.expectEqualStrings("runtime error", result.get("status").?.string);
+    try std.testing.expectEqualStrings(unsupported_metadata_freshness_message, result.get("error").?.string);
+    try std.testing.expect(result.get("criteria") == null);
+}
+
 test "sources writer emits dbt v3 success shape" {
     const source = SourceDef{
         .package_name = "demo",

tests/test_cli.py

@@ -11216,7 +11216,7 @@ def test_source_freshness_loaded_at_query_owns_filtering(tmp_path: Path):
 
 
 @pytest.mark.skipif(DUCKDB is None, reason="duckdb CLI is required for the M3 source freshness runtime-error coverage")
-def test_source_freshness_writes_runtime_error_result_for_missing_loaded_at_field(tmp_path: Path):
+def test_source_freshness_reports_unsupported_duckdb_metadata_and_continues(tmp_path: Path):
     project = copy_fixture(tmp_path, "source_freshness")
     target = tmp_path / "freshness-target"
     target.mkdir()
@@ -11228,7 +11228,12 @@ def test_source_freshness_writes_runtime_error_result_for_missing_loaded_at_fiel
             "-batch",
             "-bail",
             "-c",
-            "create schema raw; create table raw.orders (order_id integer);",
+            (
+                "create schema raw; "
+                "create table raw.customers as select 1 as customer_id, current_timestamp - interval '2 hours' as loaded_at; "
+                "create table raw.orders (order_id integer); "
+                "create table raw.query_customers as select 1 as customer_id, current_timestamp - interval '2 hours' as loaded_at;"
+            ),
         ],
         cwd=ROOT,
         check=True,
@@ -11246,26 +11251,34 @@ def test_source_freshness_writes_runtime_error_result_for_missing_loaded_at_fiel
             "--target-path",
             str(target),
             "--select",
-            "source:raw.orders",
+            "source:raw.customers source:raw.orders source:raw.query_customers",
         ],
         cwd=ROOT,
         text=True,
         capture_output=True,
     )
     assert result.returncode == 1
-    assert "Checked freshness for 1 source(s)" in result.stdout
+    assert "Checked freshness for 3 source(s)" in result.stdout
     assert "one or more source freshness checks failed" in result.stderr
 
     sources_path = target / "sources.json"
     assert_sources_schema_slice(sources_path)
     sources = json.loads(sources_path.read_text())
-    assert len(sources["results"]) == 1
-    result_row = sources["results"][0]
-    assert result_row == {
+    rows = {row["unique_id"]: row for row in sources["results"]}
+    assert sorted(rows) == [
+        "source.source_freshness.raw.customers",
+        "source.source_freshness.raw.orders",
+        "source.source_freshness.raw.query_customers",
+    ]
+    assert rows["source.source_freshness.raw.customers"]["status"] == "warn"
+    assert rows["source.source_freshness.raw.query_customers"]["status"] == "warn"
+    assert rows["source.source_freshness.raw.orders"] == {
         "unique_id": "source.source_freshness.raw.orders",
-        "error": "source freshness currently requires loaded_at_field or loaded_at_query",
+        "error": "source freshness requires loaded_at_field or loaded_at_query because the DuckDB adapter does not support metadata-based freshness",
         "status": "runtime error",
     }
+    assert "error" not in rows["source.source_freshness.raw.customers"]
+    assert "error" not in rows["source.source_freshness.raw.query_customers"]
     assert str(project) not in sources_path.read_text()