Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -515,6 +515,6 @@ Exit criteria:
- M0 is under PR review as the Zig `0.16.0` runtime scaffold.
- M1 has started on stacked branches with native Zig artifact-first parser slices.
- `dxt parse` now targets the supported Tier 0 subset: project name/model paths/seed paths/macro paths, SQL model discovery, CSV seed discovery, source discovery, exposure discovery, project macro discovery, macro property YAML for descriptions and arguments, docs block discovery, literal `ref` to models or seeds, literal `source`, literal `doc` in descriptions, inline `config(materialized=..., tags=...)`, known project macro call dependencies, narrow YAML model properties for scalar descriptions, simple columns, tags, materialization, disabled SQL models, dbt-shaped `unique`, `not_null`, `accepted_values`, and `relationships` generic test nodes, dependency maps, and deterministic partial `manifest.json`. YAML generic test arguments are currently supported for scalar values plus inline and block lists required by public Jaffle Shop DuckDB-style tests.
- `dxt ls` now lists dbt-selectable resources from the same parser graph with stable text/JSON output and basic name, tag, path, resource type including exposures, leading/trailing `+` graph expansion, and exact exclude filters; macros are emitted in artifacts but not exposed as `ls` resources.
- Synthetic fixtures cover one model, model refs, seed refs, source refs, exposure refs to models and sources, combined source/model YAML, inline config/tag selection, YAML model properties and columns, emitted `unique`, `not_null`, `accepted_values`, and `relationships` generic test nodes, project macro artifacts and macro properties, configured `macro-paths` replacing the default macro directory, macro calls recorded in model and macro `depends_on.macros`, docs blocks with literal `doc` descriptions, disabled models, disabled ref diagnostics, unmatched model-property warnings, duplicate model and docs diagnostics, unsupported dynamic ref/doc diagnostics, missing doc diagnostics, malformed docs block diagnostics, and unsupported unknown macro-call diagnostics.
- `dxt ls` now lists dbt-selectable resources from the same parser graph with stable text/JSON output and basic name, tag, path, config materialization, comma intersection, resource type including exposures, leading/trailing `+` graph expansion, and exact exclude filters; macros are emitted in artifacts but not exposed as `ls` resources.
- Synthetic fixtures cover one model, model refs, seed refs, source refs, exposure refs to models and sources, combined source/model YAML, inline config/tag selection, config materialization selection, comma-intersection selection, YAML model properties and columns, emitted `unique`, `not_null`, `accepted_values`, and `relationships` generic test nodes, project macro artifacts and macro properties, configured `macro-paths` replacing the default macro directory, macro calls recorded in model and macro `depends_on.macros`, docs blocks with literal `doc` descriptions, disabled models, disabled ref diagnostics, unmatched model-property warnings, duplicate model and docs diagnostics, unsupported dynamic ref/doc diagnostics, missing doc diagnostics, malformed docs block diagnostics, and unsupported unknown macro-call diagnostics.
- The current M1 manual gate parses the public Jaffle Shop DuckDB project into a partial manifest with SQL models, CSV seeds, docs blocks, project macros, and supported generic test nodes, including the Jaffle `accepted_values` and `relationships` tests. The next M1 slices should add published schema validation, broader selector parity, package macro namespaces, and deeper Jaffle artifact parity.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ Implemented pre-alpha commands:
./zig-out/bin/dxt ls --project-dir tests/fixtures/model_ref --output json
```

`parse` and `ls` currently support only the documented M1 parser subset: `dbt_project.yml` name/model paths/seed paths/macro paths/target path, SQL model discovery, CSV seed discovery, source discovery, exposure discovery, project macro discovery, docs block discovery, literal `ref` to models or seeds, literal `source`, literal `doc` in descriptions, basic inline `config`, narrow YAML model and macro properties, simple columns, tags, materialization and disabled SQL models, dbt-shaped generic test nodes for `unique`, `not_null`, `accepted_values`, and `relationships`, deterministic partial `manifest.json`, and basic name/tag/path/resource/graph selectors. `compile`, `build`, and `docs generate` remain planned placeholders.
`parse` and `ls` currently support only the documented M1 parser subset: `dbt_project.yml` name/model paths/seed paths/macro paths/target path, SQL model discovery, CSV seed discovery, source discovery, exposure discovery, project macro discovery, docs block discovery, literal `ref` to models or seeds, literal `source`, literal `doc` in descriptions, basic inline `config`, narrow YAML model and macro properties, simple columns, tags, materialization and disabled SQL models, dbt-shaped generic test nodes for `unique`, `not_null`, `accepted_values`, and `relationships`, deterministic partial `manifest.json`, and basic name/tag/path/resource/config materialization selectors with comma intersections and graph expansion. `compile`, `build`, and `docs generate` remain planned placeholders.

## Development

Expand Down
95 changes: 75 additions & 20 deletions src/project.zig
Original file line number Diff line number Diff line change
Expand Up @@ -1999,11 +1999,22 @@ fn matchesResourceType(requested: ?[]const u8, actual: []const u8) bool {
fn matchesSelector(graph: *const Graph, node: *const Node, spec: SelectorSpec) bool {
if (!spec.active) return true;
if (spec.value.len == 0) return true;
if (matchesNodeSelectorDirect(node, spec.value)) return true;
return matchesGraphExpansion(graph, node.unique_id, spec);
return matchesNodeSelectorExpression(graph, node, spec.value);
}

fn matchesNodeSelectorDirect(node: *const Node, value: []const u8) bool {
fn matchesNodeSelectorExpression(graph: *const Graph, node: *const Node, value: []const u8) bool {
var raw_terms = std.mem.splitScalar(u8, value, ',');
var matched_any = false;
while (raw_terms.next()) |raw_term| {
const term = parseSelectorTerm(raw_term);
if (term.value.len == 0) return false;
if (!matchesNodeSelectorTerm(node, term.value) and !matchesGraphExpansion(graph, node.unique_id, term)) return false;
matched_any = true;
}
return matched_any;
}

fn matchesNodeSelectorTerm(node: *const Node, value: []const u8) bool {
if (std.mem.eql(u8, value, node.name) or std.mem.eql(u8, value, node.unique_id)) return true;
if (std.mem.startsWith(u8, value, "tag:")) {
const tag = value["tag:".len..];
Expand All @@ -2018,17 +2029,32 @@ fn matchesNodeSelectorDirect(node: *const Node, value: []const u8) bool {
if (std.mem.startsWith(u8, value, "source:")) {
return false;
}
if (std.mem.startsWith(u8, value, "config.materialized:")) {
const materialized = value["config.materialized:".len..];
return std.mem.eql(u8, node.resource_type, "model") and std.mem.eql(u8, materialized, node.materialized);
}
return false;
}

fn matchesTestSelector(graph: *const Graph, test_node: *const GenericTestNode, spec: SelectorSpec) bool {
if (!spec.active) return true;
if (spec.value.len == 0) return true;
if (matchesTestSelectorDirect(test_node, spec.value)) return true;
return matchesGraphExpansion(graph, test_node.unique_id, spec);
return matchesTestSelectorExpression(graph, test_node, spec.value);
}

fn matchesTestSelectorDirect(test_node: *const GenericTestNode, value: []const u8) bool {
fn matchesTestSelectorExpression(graph: *const Graph, test_node: *const GenericTestNode, value: []const u8) bool {
var raw_terms = std.mem.splitScalar(u8, value, ',');
var matched_any = false;
while (raw_terms.next()) |raw_term| {
const term = parseSelectorTerm(raw_term);
if (term.value.len == 0) return false;
if (!matchesTestSelectorTerm(test_node, term.value) and !matchesGraphExpansion(graph, test_node.unique_id, term)) return false;
matched_any = true;
}
return matched_any;
}

fn matchesTestSelectorTerm(test_node: *const GenericTestNode, value: []const u8) bool {
if (std.mem.eql(u8, value, test_node.name) or std.mem.eql(u8, value, test_node.unique_id)) return true;
if (std.mem.startsWith(u8, value, "path:")) {
const path = value["path:".len..];
Expand All @@ -2040,11 +2066,22 @@ fn matchesTestSelectorDirect(test_node: *const GenericTestNode, value: []const u
fn matchesSourceSelector(graph: *const Graph, source: *const SourceDef, spec: SelectorSpec) bool {
if (!spec.active) return true;
if (spec.value.len == 0) return true;
if (matchesSourceSelectorDirect(source, spec.value)) return true;
return matchesGraphExpansion(graph, source.unique_id, spec);
return matchesSourceSelectorExpression(graph, source, spec.value);
}

fn matchesSourceSelectorExpression(graph: *const Graph, source: *const SourceDef, value: []const u8) bool {
var raw_terms = std.mem.splitScalar(u8, value, ',');
var matched_any = false;
while (raw_terms.next()) |raw_term| {
const term = parseSelectorTerm(raw_term);
if (term.value.len == 0) return false;
if (!matchesSourceSelectorTerm(source, term.value) and !matchesGraphExpansion(graph, source.unique_id, term)) return false;
matched_any = true;
}
return matched_any;
}

fn matchesSourceSelectorDirect(source: *const SourceDef, value: []const u8) bool {
fn matchesSourceSelectorTerm(source: *const SourceDef, value: []const u8) bool {
if (std.mem.eql(u8, value, source.unique_id) or std.mem.eql(u8, value, source.table_name)) return true;
if (std.mem.startsWith(u8, value, "source:")) {
const source_value = value["source:".len..];
Expand All @@ -2059,12 +2096,22 @@ fn matchesSourceSelectorDirect(source: *const SourceDef, value: []const u8) bool
fn matchesExposureSelector(graph: *const Graph, exposure: *const ExposureDef, spec: SelectorSpec) bool {
if (!spec.active) return true;
if (spec.value.len == 0) return true;
const direct = matchesExposureSelectorDirect(exposure, spec.value);
if (direct) return true;
return matchesGraphExpansion(graph, exposure.unique_id, spec);
return matchesExposureSelectorExpression(graph, exposure, spec.value);
}

fn matchesExposureSelectorDirect(exposure: *const ExposureDef, value: []const u8) bool {
fn matchesExposureSelectorExpression(graph: *const Graph, exposure: *const ExposureDef, value: []const u8) bool {
var raw_terms = std.mem.splitScalar(u8, value, ',');
var matched_any = false;
while (raw_terms.next()) |raw_term| {
const term = parseSelectorTerm(raw_term);
if (term.value.len == 0) return false;
if (!matchesExposureSelectorTerm(exposure, term.value) and !matchesGraphExpansion(graph, exposure.unique_id, term)) return false;
matched_any = true;
}
return matched_any;
}

fn matchesExposureSelectorTerm(exposure: *const ExposureDef, value: []const u8) bool {
if (std.mem.eql(u8, value, exposure.name) or std.mem.eql(u8, value, exposure.unique_id)) return true;
if (std.mem.startsWith(u8, value, "exposure:")) {
const exposure_value = value["exposure:".len..];
Expand All @@ -2087,31 +2134,39 @@ fn parseSelectorSpec(selector: ?[]const u8) SelectorSpec {
const raw = selector orelse return .{};
return .{
.active = true,
.value = trimPlus(raw),
.include_parents = std.mem.startsWith(u8, raw, "+"),
.include_children = std.mem.endsWith(u8, raw, "+"),
.value = raw,
};
}

fn parseSelectorTerm(raw: []const u8) SelectorSpec {
const trimmed = std.mem.trim(u8, raw, " \t\r");
return .{
.active = true,
.value = trimPlus(trimmed),
.include_parents = std.mem.startsWith(u8, trimmed, "+"),
.include_children = std.mem.endsWith(u8, trimmed, "+"),
};
}

fn matchesGraphExpansion(graph: *const Graph, candidate_unique_id: []const u8, spec: SelectorSpec) bool {
if (!spec.include_parents and !spec.include_children) return false;
for (graph.nodes.items) |*target| {
if (!target.enabled or !matchesNodeSelectorDirect(target, spec.value)) continue;
if (!target.enabled or !matchesNodeSelectorTerm(target, spec.value)) continue;
if (spec.include_parents and resourceDependsOn(graph, target.unique_id, candidate_unique_id)) return true;
if (spec.include_children and resourceDependsOn(graph, candidate_unique_id, target.unique_id)) return true;
}
for (graph.tests.items) |*target| {
if (!matchesTestSelectorDirect(target, spec.value)) continue;
if (!matchesTestSelectorTerm(target, spec.value)) continue;
if (spec.include_parents and resourceDependsOn(graph, target.unique_id, candidate_unique_id)) return true;
if (spec.include_children and resourceDependsOn(graph, candidate_unique_id, target.unique_id)) return true;
}
for (graph.sources.items) |*target| {
if (!matchesSourceSelectorDirect(target, spec.value)) continue;
if (!matchesSourceSelectorTerm(target, spec.value)) continue;
if (spec.include_children and resourceDependsOn(graph, candidate_unique_id, target.unique_id)) return true;
}
for (graph.exposures.items) |*target| {
if (!target.enabled) continue;
if (!matchesExposureSelectorDirect(target, spec.value)) continue;
if (!matchesExposureSelectorTerm(target, spec.value)) continue;
if (spec.include_parents and resourceDependsOn(graph, target.unique_id, candidate_unique_id)) return true;
if (spec.include_children and resourceDependsOn(graph, candidate_unique_id, target.unique_id)) return true;
}
Expand Down
38 changes: 30 additions & 8 deletions src/root.zig
Original file line number Diff line number Diff line change
Expand Up @@ -207,16 +207,38 @@ fn parseOptions(args: []const []const u8, stderr: *Io.Writer, mode: OptionMode)
}

fn validateSelector(value: []const u8) !void {
const trimmed = std.mem.trim(u8, value, "+");
if (std.mem.indexOfScalar(u8, trimmed, ':')) |_| {
if (!(std.mem.startsWith(u8, trimmed, "tag:") or
std.mem.startsWith(u8, trimmed, "path:") or
std.mem.startsWith(u8, trimmed, "source:") or
std.mem.startsWith(u8, trimmed, "exposure:")))
{
return error.UnsupportedSelector;
if (value.len == 0) return error.UnsupportedSelector;
var terms = std.mem.splitScalar(u8, value, ',');
while (terms.next()) |raw_term| {
if (raw_term.len == 0) return error.UnsupportedSelector;
const leading_plus = raw_term[0] == '+';
const trailing_plus = raw_term[raw_term.len - 1] == '+';
const start: usize = if (leading_plus) 1 else 0;
const end: usize = if (trailing_plus) raw_term.len - 1 else raw_term.len;
if (start >= end) return error.UnsupportedSelector;
const part = raw_term[start..end];
if (part.len == 0) return error.UnsupportedSelector;
if (std.mem.indexOfAny(u8, part, " \t\r")) |_| return error.UnsupportedSelector;
if (std.mem.indexOfScalar(u8, part, '+')) |_| return error.UnsupportedSelector;
if (std.mem.indexOfScalar(u8, part, ':')) |_| try validateSelectorMethod(part);
}
}

fn validateSelectorMethod(part: []const u8) !void {
const prefixes = [_][]const u8{
"tag:",
"path:",
"source:",
"exposure:",
"config.materialized:",
};
inline for (prefixes) |prefix| {
if (std.mem.startsWith(u8, part, prefix)) {
if (part.len == prefix.len) return error.UnsupportedSelector;
return;
}
}
return error.UnsupportedSelector;
}

fn requiresValue(arg: []const u8, mode: OptionMode) bool {
Expand Down
2 changes: 2 additions & 0 deletions tests/fixtures/selector_graph/models/orders.sql
Original file line number Diff line number Diff line change
@@ -1 +1,3 @@
{{ config(materialized="table") }}

select customer_id from {{ ref("customers") }}
69 changes: 61 additions & 8 deletions tests/test_cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -812,6 +812,39 @@ def test_ls_text_json_and_tag_selection(tmp_path: Path):
assert excluded.stdout == ""


def test_ls_config_materialized_and_comma_intersection(tmp_path: Path):
project = copy_fixture(tmp_path, "inline_config")

def ls_text(*args: str) -> list[str]:
result = subprocess.run(
[DXT, "ls", "--project-dir", str(project), *args],
cwd=ROOT,
text=True,
capture_output=True,
)
assert result.returncode == 0, result.stderr
return result.stdout.splitlines()

assert ls_text("--select", "config.materialized:table") == ["model.inline_config.orders"]
assert ls_text("--select", "config.materialized:view") == []
assert ls_text("--select", "tag:nightly,config.materialized:table") == [
"model.inline_config.orders"
]
assert ls_text("--select", "tag:nightly,config.materialized:view") == []
assert ls_text("--select", "orders,config.materialized:table") == ["model.inline_config.orders"]
assert ls_text("--select", "tag:nightly", "--exclude", "orders,config.materialized:table") == []

default_project = copy_fixture(tmp_path, "single_model")
default_result = subprocess.run(
[DXT, "ls", "--project-dir", str(default_project), "--select", "config.materialized:view"],
cwd=ROOT,
text=True,
capture_output=True,
)
assert default_result.returncode == 0, default_result.stderr
assert default_result.stdout.splitlines() == ["model.single_model.customers"]


def test_ls_graph_plus_selectors(tmp_path: Path):
project = copy_fixture(tmp_path, "selector_graph")

Expand Down Expand Up @@ -846,6 +879,16 @@ def ls_json(*args: str) -> list[str]:
"model.selector_graph.orders",
"model.selector_graph.stg_customers",
]
assert ls_json("--select", "customers+,config.materialized:view") == [
"model.selector_graph.customers"
]
assert ls_json("--select", "customers+,config.materialized:table") == [
"model.selector_graph.orders"
]
assert ls_json("--select", "+orders,config.materialized:view") == [
"model.selector_graph.customers",
"model.selector_graph.stg_customers",
]


def test_ls_rejects_unsupported_resource_type_and_selector(tmp_path: Path):
Expand All @@ -859,14 +902,24 @@ def test_ls_rejects_unsupported_resource_type_and_selector(tmp_path: Path):
assert unsupported_type.returncode == 2
assert "--resource-type supports only model, seed, source, exposure, or test" in unsupported_type.stderr

unsupported_selector = subprocess.run(
[DXT, "ls", "--project-dir", str(project), "--select", "config.materialized:view"],
cwd=ROOT,
text=True,
capture_output=True,
)
assert unsupported_selector.returncode == 2
assert "selector syntax is not supported" in unsupported_selector.stderr
for selector in [
"state:modified",
"config.schema:audit",
"tag:nightly,",
"config.materialized:",
"tag:nightly, config.materialized:view",
"++customers",
"customers++",
"++customers++",
]:
unsupported_selector = subprocess.run(
[DXT, "ls", "--project-dir", str(project), "--select", selector],
cwd=ROOT,
text=True,
capture_output=True,
)
assert unsupported_selector.returncode == 2
assert "selector syntax is not supported" in unsupported_selector.stderr


def test_dynamic_ref_fails_loudly(tmp_path: Path):
Expand Down
Loading