Handle trailing comments in DuckDB run SQL

Author: sabino

.agent/research/m3-duckdb-run-sql-models.md

@@ -35,6 +35,10 @@ dbt Core v1:
   `core/dbt/parser/unit_tests.py` references to `macro.dbt.is_incremental`:
   incremental models commonly include `is_incremental()` in parseable SQL even
   when this slice still rejects incremental materialization before execution.
+- `core/dbt/graph/selector.py::get_graph_queue`,
+  `core/dbt/graph/queue.py::GraphQueue`, and
+  `core/dbt/task/runnable.py::{get_graph_queue, run_queue}`: selected runnable
+  nodes execute through a dependency-aware graph queue.
 
 dbt Core v2 / Fusion:
 
@@ -58,14 +62,21 @@ dbt Core v2 / Fusion:
   Fusion carries `is_incremental()` as a dbt macro. dxt parses the zero-argument
   call narrowly so run can report the unsupported incremental boundary before
   compile-time Jinja rendering.
+- `crates/dbt-dag/src/schedule.rs::Schedule` and
+  `crates/dbt-dag/src/deps_mgmt.rs::topological_sort`: Fusion represents DAG
+  scheduling and topological ordering explicitly.
 
 ## dxt Ownership
 
 - `src/project/duckdb.zig`: DuckDB CLI-backed database path resolution,
   local-file path guardrails, table/view materialization SQL rendering, and
-  execution.
+  execution. Relative DuckDB profile paths resolve from the loaded
+  `profiles.yml` directory as a deterministic dxt-local choice for this
+  CLI-backed slice; the cited Fusion auth code grounds `path` as the DuckDB
+  database option, not that relative-path base.
 - `src/project/run_results.zig`: minimal v6 `run_results.json` rendering for
-  successful SQL model runs.
+  successful SQL model runs. Failed and partial run-results artifacts are
+  planned but not implemented in this first success-only execution slice.
 - `src/project/profile.zig`: narrow scalar DuckDB `path` capture.
 - `src/project.zig`: current command facade orchestration for `run`.
 - `tests/schemas/dbt_run_results_v6_m3_slice.schema.json`: pinned schema slice
@@ -75,22 +86,29 @@ dbt Core v2 / Fusion:
 
 - Native Zig tests cover DuckDB materialization SQL rendering, conflicting
   table/view drop SQL, trailing SQL terminator trimming for wrapped model SQL,
-  unsupported materialization rejection, DuckDB profile `path` capture,
-  `is_incremental()` config scanning, and run-results JSON shape.
+  preserving non-terminated SQL trailing comments, unsupported materialization
+  rejection, DuckDB profile `path` capture, `is_incremental()` config scanning,
+  and run-results JSON shape.
 - Pytest black-box coverage executes `dxt run` against copied and generated
   fixtures, verifies dependency-order execution where lexical order conflicts
   with graph order, checks table-to-view replacement on rerun, checks
-  trailing-semicolon SQL model execution, checks profile-relative DuckDB paths,
-  queries the resulting DuckDB database through the DuckDB CLI, validates
-  `run_results.json` against the pinned schema slice, and keeps non-model, non-DuckDB,
-  unsupported-materialization, and `build` boundaries explicit.
+  trailing-semicolon SQL model execution including trailing SQL comments, checks
+  profile-relative DuckDB paths, queries the resulting DuckDB database through
+  the DuckDB CLI, validates `run_results.json` against the pinned schema slice,
+  and keeps non-model, non-DuckDB, unsupported-materialization, and `build`
+  boundaries explicit.
 
 ## Stop Conditions
 
 - Do not implement `dxt build` execution in this slice.
 - Do not implement seeds, tests, snapshots, incremental, ephemeral, Python
   models, hooks, grants, docs persistence, catalog introspection, relation
   staging/backup rename semantics, adapter caching, or threaded scheduling.
+- Do not treat direct `create or replace` as full dbt materialization parity.
+  It is a temporary shortcut for the M3 table/view success path; full parity
+  needs intermediate/backup relation handling and cleanup semantics.
+- Do not emit failed or partial `run_results.json` in this success-only slice.
+  Failure artifacts belong in the follow-up runner/task-result slice.
 - Do not call DuckDB from Python product code. Python remains a black-box test
   harness only.
 - Do not treat the external CLI backend as the long-term adapter ABI; replace it

PLAN.md

@@ -358,14 +358,15 @@ materialization execution, catalog introspection, or run-results artifacts.
 Current DuckDB SQL model run source note:
 `.agent/research/m3-duckdb-run-sql-models.md` maps dbt Core v1 Run Results v6
 schema and run-result processing plus Fusion run-results structs, task stats,
-DuckDB profile `path`, and DuckDB table/view SQL primitives to dxt's first
-execution slice. This slice lets `dxt run` execute selected enabled DuckDB SQL
-models with `table` and `view` materializations through a Zig-owned external
-DuckDB CLI backend, write compiled SQL, `manifest.json`, and a minimal
-`run_results.json`, and reject non-model selections, non-DuckDB adapters, and
-unsupported materializations explicitly. It does not implement `build`
-execution, seeds, tests, snapshots, incremental, ephemeral, hooks, grants, docs
-persistence, catalog introspection, relation staging/backup rename parity,
+DuckDB profile `path`, DuckDB table/view SQL primitives, and dbt/Fusion DAG
+queue ordering to dxt's first execution slice. This slice lets `dxt run`
+execute selected enabled DuckDB SQL models with `table` and `view`
+materializations through a Zig-owned external DuckDB CLI backend, write compiled
+SQL, `manifest.json`, and a minimal success-only `run_results.json`, and reject
+non-model selections, non-DuckDB adapters, and unsupported materializations
+explicitly. It does not implement `build` execution, seeds, tests, snapshots,
+incremental, ephemeral, hooks, grants, docs persistence, catalog introspection,
+failed or partial run-results artifacts, relation staging/backup rename parity,
 threaded scheduling, or embedded `libduckdb`.
 
 The next source-grounded M1/M2 slices after macro block variant support are:
@@ -731,7 +732,7 @@ Exit criteria:
 - `dxt ls` now lists dbt-selectable resources from the same parser graph, including scalar project/CLI var-resolved dependency edges, with stable text/JSON output and basic name/FQN wildcards, tag wildcards, slash-aware `path:` wildcards, exact `package:`/`package:this`, `source:` wildcards including package-qualified source selectors, `exposure:` wildcards, `resource_type:`, `test_type:generic`, config materialization, comma intersection, whitespace union, multi-argument selector lists, repeated selector flags, leading/trailing `+` graph expansion, and exact exclude filters; macros are emitted in artifacts but not exposed as `ls` resources.
 - `dxt compile` has started as a render-only M2 boundary for the current graph subset. It loads and resolves the same Zig parser graph, applies `--select` and `--exclude`, compiles selected enabled SQL model nodes, writes compiled SQL under `target/compiled/<package>/...`, and emits `compiled`, `compiled_code`, `compiled_path`, `relation_name`, `extra_ctes`, and `extra_ctes_injected` only for compiled model nodes. The current compiler renders `config` to empty text, literal or narrow scalar var-backed `ref`/`source` calls to deterministic quoted relation names, profile-derived `target.*`, current-model `this`, and quoted literal inline `config(schema=..., alias=...)` as default dbt relation schema/identifier components without opening a database connection.
 - `dxt docs generate` has started as an adapter-free docs artifact boundary. It loads and resolves the same Zig parser graph, applies `--select` and `--exclude` to compiled model output, writes compiled SQL, writes `manifest.json`, and writes an empty dbt-shaped `catalog.json` because adapter relation introspection is not implemented yet. Macro execution, materializations, tests, profiles-derived relation identity, adapters, `run_results.json`, non-empty `catalog.json`, and `docs serve` remain out of scope.
-- `dxt run` has started the M3 DuckDB execution path for selected enabled SQL models. It loads and resolves the same Zig parser graph, applies supported selectors/excludes, compiles selected SQL models, validates that selected models use only `table` or `view` materializations before opening DuckDB, executes selected models in dependency order through a Zig-owned external DuckDB CLI backend, writes compiled SQL, writes `manifest.json`, and writes a minimal dbt-shaped `run_results.json` v6 slice. It supports default `target/dxt.duckdb` output plus scalar DuckDB profile `path` resolved relative to the loaded `profiles.yml` directory. It does not execute seeds, tests, snapshots, incremental, ephemeral, hooks, grants, docs persistence, catalog introspection, relation staging/backup rename parity, threaded scheduling, `:memory:`, MotherDuck, or embedded `libduckdb`.
+- `dxt run` has started the M3 DuckDB execution path for selected enabled SQL models. It loads and resolves the same Zig parser graph, applies supported selectors/excludes, compiles selected SQL models, validates that selected models use only `table` or `view` materializations before opening DuckDB, executes selected models in dependency order through a Zig-owned external DuckDB CLI backend, writes compiled SQL, writes `manifest.json`, and writes a minimal dbt-shaped success-only `run_results.json` v6 slice after completed runs. It supports default `target/dxt.duckdb` output plus scalar DuckDB profile `path` resolved relative to the loaded `profiles.yml` directory as a deterministic dxt-local path-base choice for this first CLI-backed slice. It does not execute seeds, tests, snapshots, incremental, ephemeral, hooks, grants, docs persistence, catalog introspection, failure/partial run-results artifacts, relation staging/backup rename parity, threaded scheduling, `:memory:`, MotherDuck, or embedded `libduckdb`.
 - `dxt build` remains a truthful execution preflight boundary. It loads and resolves the same Zig parser graph, applies supported selectors/excludes, compiles selected SQL models where applicable, writes `manifest.json`, and then fails before seed/model/test execution with explicit runner boundary errors.
 - Synthetic fixtures cover one model, model refs, seed refs, source refs, narrow scalar var-backed model/source refs with CLI overrides and positional string defaults, exposure refs to models and sources, combined source/model YAML, inline config/tag selection, config materialization selection, comma-intersection selection, YAML model properties and columns, emitted `unique`, `not_null`, `accepted_values`, and `relationships` generic test nodes, project macro artifacts, macro block variants, macro materialization `supported_languages`, and macro properties including patched `docs` and `meta`, configured `macro-paths` replacing the default macro directory, installed package macros with package-qualified calls and package-local macro calls, installed package models, seeds, sources, docs, exposures, package YAML model properties, root package config overrides, and package-qualified/package-local refs/sources, macro calls recorded in model and macro `depends_on.macros`, docs blocks with literal `doc` descriptions, disabled models, disabled ref diagnostics, unmatched model-property warnings, duplicate model and docs diagnostics, unsupported dynamic doc diagnostics, unresolved var diagnostics for var-backed refs without scalar/default values, missing doc diagnostics, malformed docs block diagnostics, unresolved package macro diagnostics, and unsupported unknown macro-call diagnostics.
 - The committed M1 public Jaffle gate lives in `scripts/check_jaffle_shop_duckdb_parse.py`. It clones a pinned public Jaffle Shop DuckDB ref into a temporary directory by default, runs the Zig `dxt` binary, validates the current M1 manifest schema slice, asserts the supported partial manifest shape with five SQL models, three CSV seeds, two docs blocks, twenty supported generic test nodes, model/test `refs` artifact fields, dependency maps, materialization/docs config, and checks representative `dxt ls` selector behavior for resource types, materialization config, wildcards, path selectors, and graph expansion. It is a developer-side Python compatibility harness only; product parse/list behavior remains implemented in Zig. Remaining M1 work includes package-provided generic tests/macros beyond the current narrow macro call surface and deeper Jaffle artifact parity.