Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions .agent/research/dbt-upstream-reference-map.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ Every compatibility slice should record:
| Artifact schemas | `schemas/dbt/manifest/v12.json`, `schemas/dbt/run-results/v6.json`, `schemas/dbt/sources/v3.json`, `schemas/dbt/catalog/v1.json` | v2 still emits JSON for compatibility and adds Parquet artifacts per README; manifest builder in `crates/dbt-schemas/src/schemas/manifest/manifest.rs` | `src/project/manifest.zig`, future run/catalog/source writers and schema validators under tests/scripts |
| Docs catalog generation | `core/dbt/task/docs/generate.py::GenerateTask.run`, selected source handling in `_get_selected_source_ids`, `Catalog`, `Catalog.make_unique_id_map`, `build_catalog_table`, `format_stats`; `core/dbt/artifacts/schemas/catalog/v1/catalog.py::CatalogArtifact`, `CatalogResults` | `crates/dbt-schemas/src/schemas/legacy_catalog/catalog.rs::CatalogTable`, `ColumnMetadata`, `CatalogNodeStats`, `DbtCatalog`, `build_catalog`; Fusion index metadata in `crates/dbt-index-core/src/ingest/ingest_state.rs` | `src/project.zig` owns current docs orchestration; `src/project/catalog.zig` owns dbt-shaped catalog JSON for `nodes` and `sources`; `src/project/duckdb.zig` owns the first local DuckDB relation/column introspection for already-materialized selected model/seed nodes and selected source relations |
| Source freshness and `sources.json` | `core/dbt/task/freshness.py::FreshnessRunner.execute`, `FreshnessSelector.node_is_match`, `FreshnessTask.result_path`, `FreshnessTask.get_result`; `core/dbt/artifacts/schemas/freshness/v3/freshness.py::FreshnessExecutionResultArtifact`, `SourceFreshnessOutput`, `SourceFreshnessRuntimeError`; `core/dbt/parser/sources.py::SourceParser.parse_source`, `calculate_loaded_at_field_query_from_raw_target`, `merge_source_freshness`; `schemas/dbt/sources/v3.json` | `crates/dbt-schemas/src/schemas/sources.rs::FreshnessResultsArtifact`, `FreshnessResultsMetadata`, `FreshnessResultsNode`; `crates/dbt-scheduler/src/node_selector.rs::match_source_status`; wider parse merge references in `crates/dbt-parser/src/resolve/resolve_sources.rs` | `src/root.zig` owns `dxt source freshness` command dispatch; `src/project.zig` owns first orchestration until runner extraction; `src/project/types.zig` owns source freshness fields; `src/project/parse.zig` owns table-level YAML parsing; `src/project/duckdb.zig` owns DuckDB loaded-at-field query execution; `src/project/source_freshness.zig` owns status calculation and `sources.json` v3 rendering |
| Source table identifier | `core/dbt/parser/sources.py::SourcePatcher.parse_source`, `_get_relation_name`; `core/dbt/artifacts/resources/v1/source_definition.py::ParsedSourceMandatory`; `core/dbt/context/providers.py::ParseSourceResolver.resolve`, `RuntimeSourceResolver.resolve`; `core/dbt/contracts/graph/manifest.py::SourceLookup`, `Manifest.resolve_source`; `core/dbt/graph/selector_methods.py::SourceSelectorMethod.search`; `core/dbt/task/list.py::ListTask.generate_selectors`; `schemas/dbt/manifest/v12.json` source `name` and `identifier` fields | `crates/dbt-parser/src/resolve/resolve_sources.rs::resolve_sources`; `crates/dbt-schemas/src/schemas/manifest/manifest_nodes.rs::ManifestSource`; `crates/dbt-schemas/src/schemas/nodes.rs::DbtSourceAttr`, `DbtSource::search_name`, `DbtSource::selector_string`; `crates/dbt-parser/src/resolve/resolve_tests/persist_generic_data_tests.rs::format_node_unique_id` | `src/project/types.zig` owns `SourceDef.identifier`; `src/project/parse.zig` parses table-level YAML `identifier`; `src/project/compiler.zig` owns source physical relation rendering; `src/project/manifest.zig` emits source `identifier` and `relation_name`; `src/project/duckdb.zig` uses physical source identifiers for catalog, freshness, and source generic-test relation SQL while logical selectors/deps stay unchanged |
| Command surface | dbt v1 command behavior through parser/runner contracts and artifacts | `crates/dbt-clap-core/src/commands.rs::CoreCommand`, static-analysis flags and command parsing | `src/root.zig`, `src/main.zig`, future command-specific modules |
| Adapter capability and SQL identity | v1 adapter behavior is distributed across adapters and context providers | `crates/dbt-adapter-core/src/lib.rs::AdapterType`, `quote_char`, static-analysis support matrix, microbatch capability; `crates/dbt-adapter-sql/src/ident.rs`, `statements.rs`, `types/*` | Future `src/project/adapter.zig`, `src/project/sql.zig`, and cross-database planner modules |
| DuckDB SQL model execution and run results | `schemas/dbt/run-results/v6.json`; `core/dbt/artifacts/schemas/run/v5/run.py::RunResultOutput`, `process_run_result`, `RunResultsArtifact.from_execution_results`; `core/dbt/compilation.py::Compiler.compile_node`, `write_graph_file` | `crates/dbt-auth/src/duckdb/mod.rs::DuckDbAuth.configure`; `crates/dbt-loader/src/dbt_macro_assets/dbt-duckdb/macros/adapters.sql::duckdb__create_table_as`, `duckdb__create_view_as`; `crates/dbt-loader/src/dbt_macro_assets/dbt-duckdb/macros/materializations/table.sql`; `crates/dbt-loader/src/dbt_macro_assets/dbt-adapters/macros/materializations/models/view.sql`; `crates/dbt-schemas/src/schemas/run_results.rs::RunResultOutput`, `RunResultsArtifact`; `crates/dbt-tasks-core/src/stats_to_results.rs`, `utils.rs::build_run_results_artifact` | `src/project/duckdb.zig` owns the first CLI-backed DuckDB execution slice, local-file path guardrails, and table/view SQL rendering; `src/project.zig` currently owns selected-model dependency ordering until a runner module exists; `src/project/run_results.zig` owns the minimal v6 run-results writer; future adapter ABI should replace the CLI backend with embedded DuckDB/linking and add task timing, adapter responses, relation staging, DAG scheduling, seeds, and tests |
Expand Down Expand Up @@ -93,9 +94,11 @@ Every compatibility slice should record:
when a selected source is stale past `error_after` or has a runtime error.
Empty or all-null loaded-at values are emitted as stale freshness results.
Source/table `config:` inheritance, dbt-shaped freshness threshold merging,
final `freshness: null`, and narrow source schema rendering are documented in
`.agent/research/m2-source-config-freshness-inheritance.md`. Jinja rendering
inside `loaded_at_query`, metadata freshness, source-status selectors, hooks,
final `freshness: null`, narrow source schema rendering, and source table
`identifier` physical-name overrides are documented in
`.agent/research/m2-source-config-freshness-inheritance.md` and
`.agent/research/m2-source-table-identifier.md`. Jinja rendering inside
`loaded_at_query`, metadata freshness, source-status selectors, hooks,
threaded scheduling, non-DuckDB adapters, and embedded `libduckdb` remain
future work.
- `src/project/loader.zig` now owns graph loading order, installed-package
Expand Down
60 changes: 60 additions & 0 deletions .agent/research/m2-source-table-identifier.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# M2 Source Table Identifier Slice

## Scope

Add dbt source table `identifier` support as a narrow physical-relation-name
override. The logical source table `name` remains the key used by
`source(source_name, table_name)`, unique IDs, selectors, dependency maps, and
artifact `name` / `fqn` fields. The optional table `identifier` controls the
rendered relation identifier, manifest `identifier`, DuckDB source catalog
lookup, source freshness SQL, and source generic-test SQL.

## Upstream References

- dbt Core v1:
- `core/dbt/parser/sources.py::SourcePatcher.parse_source` defaults source
table `identifier` from table `name` while keeping logical `name` separate.
- `core/dbt/parser/sources.py::SourcePatcher._get_relation_name` builds the
rendered relation name from source relation components.
- `core/dbt/contracts/graph/nodes.py::SourceDefinition.same_database_representation`
treats `identifier` as part of source relation identity.
- `core/dbt/context/providers.py::ParseSourceResolver.resolve` and
`RuntimeSourceResolver.resolve`, plus
`core/dbt/contracts/graph/manifest.py::Manifest.resolve_source`, resolve
by logical source/table names.
- `schemas/dbt/manifest/v12.json` requires source `identifier` separately
from source `name`.
- dbt Core v2 / Fusion:
- `crates/dbt-parser/src/resolve/resolve_sources.rs::resolve_sources`
defaults table `identifier` from table `name`, preserves raw configured
identifiers for source attributes, normalizes relation components, and
builds relation names from database/schema/identifier.
- `crates/dbt-schemas/src/schemas/nodes.rs::DbtSourceAttr` stores
`identifier` separately from `source_name`.

## dxt Ownership

- `src/project/types.zig`: `SourceDef.identifier`.
- `src/project/parse.zig`: table-level YAML `identifier` parsing.
- `src/project/compiler.zig`: `sourceIdentifier` and `relationNameForSource`.
- `src/project/manifest.zig`: source `identifier` and `relation_name` fields.
- `src/project/duckdb.zig`: source freshness, catalog, and source generic-test
relation rendering.

## Validation

- Native Zig parser test for logical table name plus physical identifier.
- Native compiler, manifest, and DuckDB tests proving relation rendering uses
the configured identifier.
- Python CLI tests proving compile/manifest/selector behavior preserves logical
source keys while rendered SQL, catalog, and source freshness use the physical
identifier.

## Stop Conditions

- Do not implement source `database`, `quoting`, project-level source config, or
metadata freshness in this slice.
- Do not change selector semantics or source unique IDs from logical table
names.
- Do not add general Jinja rendering for source properties.
- Product behavior remains Zig; Python is only black-box CLI/artifact coverage.
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@ compatibility.
freshness inheritance, narrow source `schema: "{{ target.schema }}_raw"`
rendering, expanded Manifest v12-shaped source fields, and DuckDB source
freshness execution against resolved inherited source settings.
- Source table `identifier` parsing as a physical relation-name override for
`source()` compilation, manifest source fields, DuckDB docs catalog lookup,
source freshness SQL, and source generic-test relation rendering while
preserving logical source selectors and unique IDs.
- `file:` selector support for basename/stem matching across selectable graph
resources, with selector reuse covered through `ls` and `docs generate`.
- Depth-limited dbt-style `+` graph selectors for parent and child expansion,
Expand Down
Loading
Loading