Skip to content

Commit b5199e9

Browse files
sabinosabino
authored
Extract duplicate validation helpers (#40)
1 parent 597041e commit b5199e9

3 files changed

Lines changed: 123 additions & 81 deletions

File tree

‎PLAN.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -569,7 +569,7 @@ Exit criteria:
569569

570570
- M0 is complete as the Zig `0.16.0` runtime scaffold.
571571
- M1 has started on stacked branches with native Zig artifact-first parser slices.
572-
- M1A has started with behavior-preserving `src/project/types.zig`, `src/project/selector.zig`, `src/project/manifest.zig`, `src/project/util.zig`, `src/project/config.zig`, `src/project/fs.zig`, `src/project/jinja.zig`, `src/project/resolve.zig`, and `src/project/parse.zig` extractions. `src/project/manifest.zig` owns selected-resource JSON and partial `manifest.json` writing with native tests for selected JSON shape, JSON escaping, exposure dependency ordering, disabled-resource filtering, and graph-map output. `src/project/util.zig` owns shared display, membership, append-dedup, string sorting, and narrow YAML scalar/list helpers. `src/project/config.zig` owns `dbt_project.yml` loading and project path/docs config parsing. `src/project/fs.zig` owns deterministic resource file discovery and Linux directory traversal helpers. `src/project/jinja.zig` owns lexical Jinja call, parenthesis, quoted string, and literal argument helpers while graph-aware scanning remains in `src/project.zig`. `src/project/resolve.zig` owns graph lookup/count helpers, low-level ref/source resolution helpers, and dependency-map mutation for refs, sources, and known macro dependencies. `src/project/parse.zig` owns narrow parser scalar helpers for YAML booleans, JSON-compatible scalar classification, generic-test YAML item names, generic-test definition construction/cloning, and generic-test identity/name/hash helpers with native tests. `src/project.zig` remains the public parser/list facade and still owns most loader, resource parser, scanner, and resolver orchestration code until follow-up extractions move those pieces behind focused internal modules.
572+
- M1A has started with behavior-preserving `src/project/types.zig`, `src/project/selector.zig`, `src/project/manifest.zig`, `src/project/util.zig`, `src/project/config.zig`, `src/project/fs.zig`, `src/project/jinja.zig`, `src/project/resolve.zig`, and `src/project/parse.zig` extractions. `src/project/manifest.zig` owns selected-resource JSON and partial `manifest.json` writing with native tests for selected JSON shape, JSON escaping, exposure dependency ordering, disabled-resource filtering, and graph-map output. `src/project/util.zig` owns shared display, membership, append-dedup, string sorting, and narrow YAML scalar/list helpers. `src/project/config.zig` owns `dbt_project.yml` loading and project path/docs config parsing. `src/project/fs.zig` owns deterministic resource file discovery and Linux directory traversal helpers. `src/project/jinja.zig` owns lexical Jinja call, parenthesis, quoted string, and literal argument helpers while graph-aware scanning remains in `src/project.zig`. `src/project/resolve.zig` owns graph lookup/count helpers, duplicate resource validation, low-level ref/source resolution helpers, and dependency-map mutation for refs, sources, and known macro dependencies. `src/project/parse.zig` owns narrow parser scalar helpers for YAML booleans, JSON-compatible scalar classification, generic-test YAML item names, generic-test definition construction/cloning, and generic-test identity/name/hash helpers with native tests. `src/project.zig` remains the public parser/list facade and still owns most loader, resource parser, scanner, and resolver orchestration code until follow-up extractions move those pieces behind focused internal modules.
573573
- `dxt parse` now targets the supported Tier 0 subset: project name/model paths/seed paths/macro paths, project and package model path configs for literal `+materialized`, `+tags`, and model/seed `+docs.node_color`, root-project model config overrides for installed packages, SQL model discovery, CSV seed discovery, installed package SQL model and CSV seed discovery from `dbt_packages`, source discovery, installed package source discovery, exposure discovery, installed package exposure discovery, project macro discovery, installed package macro discovery from `dbt_packages`, macro property YAML for project macro descriptions and arguments, project and package docs block discovery, literal `ref` to models or seeds, two-argument package refs, package-local refs in installed package models and exposures, unique installed-package fallback for unqualified refs, literal `source`, package-local sources in installed package models, unique installed-package fallback for unqualified sources, literal `doc` in project and package descriptions, inline `config(materialized=..., tags=...)`, known project/package-qualified/package-local macro call dependencies, narrow project and package YAML model properties for scalar descriptions, simple columns, tags, materialization, disabled SQL models, dbt-shaped `unique`, `not_null`, `accepted_values`, and `relationships` generic test nodes, model/test `refs` and `sources` artifact fields, dependency maps, and deterministic partial `manifest.json`. The manifest includes the v12 top-level maps needed by the M1 artifact shape and is covered by a pinned local dbt Manifest v12 schema slice. YAML generic test arguments are currently supported for scalar values plus inline and block lists required by public Jaffle Shop DuckDB-style tests.
574574
- `dxt ls` now lists dbt-selectable resources from the same parser graph with stable text/JSON output and basic name/FQN wildcards, tag wildcards, slash-aware `path:` wildcards, exact `package:`/`package:this`, `source:` wildcards including package-qualified source selectors, `exposure:` wildcards, `resource_type:`, `test_type:generic`, config materialization, comma intersection, whitespace union, multi-argument selector lists, repeated selector flags, leading/trailing `+` graph expansion, and exact exclude filters; macros are emitted in artifacts but not exposed as `ls` resources.
575575
- Synthetic fixtures cover one model, model refs, seed refs, source refs, exposure refs to models and sources, combined source/model YAML, inline config/tag selection, config materialization selection, comma-intersection selection, YAML model properties and columns, emitted `unique`, `not_null`, `accepted_values`, and `relationships` generic test nodes, project macro artifacts and macro properties, configured `macro-paths` replacing the default macro directory, installed package macros with package-qualified calls and package-local macro calls, installed package models, seeds, sources, docs, exposures, package YAML model properties, root package config overrides, and package-qualified/package-local refs/sources, macro calls recorded in model and macro `depends_on.macros`, docs blocks with literal `doc` descriptions, disabled models, disabled ref diagnostics, unmatched model-property warnings, duplicate model and docs diagnostics, unsupported dynamic ref/doc diagnostics, missing doc diagnostics, malformed docs block diagnostics, unresolved package macro diagnostics, and unsupported unknown macro-call diagnostics.

‎src/project.zig‎

Lines changed: 6 additions & 80 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,12 @@ const findMacroIdForUnqualifiedCall = project_resolve.findMacroIdForUnqualifiedC
7474
const findMacroIndexByPackageAndName = project_resolve.findMacroIndexByPackageAndName;
7575
const findModelIndexByName = project_resolve.findModelIndexByName;
7676
const hasMacroPackage = project_resolve.hasMacroPackage;
77+
const rejectDuplicateDocs = project_resolve.rejectDuplicateDocs;
78+
const rejectDuplicateExposures = project_resolve.rejectDuplicateExposures;
79+
const rejectDuplicateMacroProperties = project_resolve.rejectDuplicateMacroProperties;
80+
const rejectDuplicateMacros = project_resolve.rejectDuplicateMacros;
81+
const rejectDuplicateModels = project_resolve.rejectDuplicateModels;
82+
const rejectDuplicateSeeds = project_resolve.rejectDuplicateSeeds;
7783
const resolveDependencies = project_resolve.resolveDependencies;
7884
const resolveRefDependency = project_resolve.resolveRefDependency;
7985

@@ -1209,86 +1215,6 @@ fn appendColumnClone(graph: *Graph, package_name: []const u8, columns: *std.Arra
12091215
try columns.append(graph.allocator, column);
12101216
}
12111217

1212-
fn rejectDuplicateModels(graph: *const Graph) !void {
1213-
var i: usize = 0;
1214-
while (i < graph.nodes.items.len) : (i += 1) {
1215-
var j = i + 1;
1216-
while (j < graph.nodes.items.len) : (j += 1) {
1217-
if (std.mem.eql(u8, graph.nodes.items[i].resource_type, "model") and
1218-
std.mem.eql(u8, graph.nodes.items[j].resource_type, "model") and
1219-
std.mem.eql(u8, graph.nodes.items[i].unique_id, graph.nodes.items[j].unique_id))
1220-
{
1221-
return error.DuplicateModelName;
1222-
}
1223-
}
1224-
}
1225-
}
1226-
1227-
fn rejectDuplicateSeeds(graph: *const Graph) !void {
1228-
var i: usize = 0;
1229-
while (i < graph.nodes.items.len) : (i += 1) {
1230-
var j = i + 1;
1231-
while (j < graph.nodes.items.len) : (j += 1) {
1232-
if (std.mem.eql(u8, graph.nodes.items[i].resource_type, "seed") and
1233-
std.mem.eql(u8, graph.nodes.items[j].resource_type, "seed") and
1234-
std.mem.eql(u8, graph.nodes.items[i].unique_id, graph.nodes.items[j].unique_id))
1235-
{
1236-
return error.DuplicateSeedName;
1237-
}
1238-
}
1239-
}
1240-
}
1241-
1242-
fn rejectDuplicateDocs(graph: *const Graph) !void {
1243-
var i: usize = 0;
1244-
while (i < graph.docs.items.len) : (i += 1) {
1245-
var j = i + 1;
1246-
while (j < graph.docs.items.len) : (j += 1) {
1247-
if (std.mem.eql(u8, graph.docs.items[i].unique_id, graph.docs.items[j].unique_id)) {
1248-
return error.DuplicateDocName;
1249-
}
1250-
}
1251-
}
1252-
}
1253-
1254-
fn rejectDuplicateExposures(graph: *const Graph) !void {
1255-
var i: usize = 0;
1256-
while (i < graph.exposures.items.len) : (i += 1) {
1257-
var j = i + 1;
1258-
while (j < graph.exposures.items.len) : (j += 1) {
1259-
if (std.mem.eql(u8, graph.exposures.items[i].unique_id, graph.exposures.items[j].unique_id)) {
1260-
return error.DuplicateExposureName;
1261-
}
1262-
}
1263-
}
1264-
}
1265-
1266-
fn rejectDuplicateMacroProperties(graph: *const Graph) !void {
1267-
var i: usize = 0;
1268-
while (i < graph.macro_properties.items.len) : (i += 1) {
1269-
var j = i + 1;
1270-
while (j < graph.macro_properties.items.len) : (j += 1) {
1271-
if (std.mem.eql(u8, graph.macro_properties.items[i].package_name, graph.macro_properties.items[j].package_name) and
1272-
std.mem.eql(u8, graph.macro_properties.items[i].name, graph.macro_properties.items[j].name))
1273-
{
1274-
return error.DuplicateMacroProperty;
1275-
}
1276-
}
1277-
}
1278-
}
1279-
1280-
fn rejectDuplicateMacros(graph: *const Graph) !void {
1281-
var i: usize = 0;
1282-
while (i < graph.macros.items.len) : (i += 1) {
1283-
var j = i + 1;
1284-
while (j < graph.macros.items.len) : (j += 1) {
1285-
if (std.mem.eql(u8, graph.macros.items[i].unique_id, graph.macros.items[j].unique_id)) {
1286-
return error.DuplicateMacroName;
1287-
}
1288-
}
1289-
}
1290-
}
1291-
12921218
fn resolveMacroDependencies(graph: *Graph) !void {
12931219
for (graph.macros.items) |*macro| {
12941220
try scanMacroSqlForKnownMacroCalls(graph.allocator, macro.macro_sql, graph, macro.unique_id, &macro.macro_depends_on);

‎src/project/resolve.zig‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -141,6 +141,86 @@ pub fn resolveDependencies(graph: *Graph) !void {
141141
}
142142
}
143143

144+
pub fn rejectDuplicateModels(graph: *const Graph) !void {
145+
var i: usize = 0;
146+
while (i < graph.nodes.items.len) : (i += 1) {
147+
var j = i + 1;
148+
while (j < graph.nodes.items.len) : (j += 1) {
149+
if (std.mem.eql(u8, graph.nodes.items[i].resource_type, "model") and
150+
std.mem.eql(u8, graph.nodes.items[j].resource_type, "model") and
151+
std.mem.eql(u8, graph.nodes.items[i].unique_id, graph.nodes.items[j].unique_id))
152+
{
153+
return error.DuplicateModelName;
154+
}
155+
}
156+
}
157+
}
158+
159+
pub fn rejectDuplicateSeeds(graph: *const Graph) !void {
160+
var i: usize = 0;
161+
while (i < graph.nodes.items.len) : (i += 1) {
162+
var j = i + 1;
163+
while (j < graph.nodes.items.len) : (j += 1) {
164+
if (std.mem.eql(u8, graph.nodes.items[i].resource_type, "seed") and
165+
std.mem.eql(u8, graph.nodes.items[j].resource_type, "seed") and
166+
std.mem.eql(u8, graph.nodes.items[i].unique_id, graph.nodes.items[j].unique_id))
167+
{
168+
return error.DuplicateSeedName;
169+
}
170+
}
171+
}
172+
}
173+
174+
pub fn rejectDuplicateDocs(graph: *const Graph) !void {
175+
var i: usize = 0;
176+
while (i < graph.docs.items.len) : (i += 1) {
177+
var j = i + 1;
178+
while (j < graph.docs.items.len) : (j += 1) {
179+
if (std.mem.eql(u8, graph.docs.items[i].unique_id, graph.docs.items[j].unique_id)) {
180+
return error.DuplicateDocName;
181+
}
182+
}
183+
}
184+
}
185+
186+
pub fn rejectDuplicateExposures(graph: *const Graph) !void {
187+
var i: usize = 0;
188+
while (i < graph.exposures.items.len) : (i += 1) {
189+
var j = i + 1;
190+
while (j < graph.exposures.items.len) : (j += 1) {
191+
if (std.mem.eql(u8, graph.exposures.items[i].unique_id, graph.exposures.items[j].unique_id)) {
192+
return error.DuplicateExposureName;
193+
}
194+
}
195+
}
196+
}
197+
198+
pub fn rejectDuplicateMacroProperties(graph: *const Graph) !void {
199+
var i: usize = 0;
200+
while (i < graph.macro_properties.items.len) : (i += 1) {
201+
var j = i + 1;
202+
while (j < graph.macro_properties.items.len) : (j += 1) {
203+
if (std.mem.eql(u8, graph.macro_properties.items[i].package_name, graph.macro_properties.items[j].package_name) and
204+
std.mem.eql(u8, graph.macro_properties.items[i].name, graph.macro_properties.items[j].name))
205+
{
206+
return error.DuplicateMacroProperty;
207+
}
208+
}
209+
}
210+
}
211+
212+
pub fn rejectDuplicateMacros(graph: *const Graph) !void {
213+
var i: usize = 0;
214+
while (i < graph.macros.items.len) : (i += 1) {
215+
var j = i + 1;
216+
while (j < graph.macros.items.len) : (j += 1) {
217+
if (std.mem.eql(u8, graph.macros.items[i].unique_id, graph.macros.items[j].unique_id)) {
218+
return error.DuplicateMacroName;
219+
}
220+
}
221+
}
222+
}
223+
144224
fn hasNode(graph: *const Graph, unique_id: []const u8) bool {
145225
for (graph.nodes.items) |node| {
146226
if (node.enabled and std.mem.eql(u8, node.unique_id, unique_id)) return true;
@@ -301,6 +381,42 @@ test "source resolution prefers current package and errors on ambiguous fallback
301381
try std.testing.expectError(error.UnresolvedSource, resolveSourceDependency(&graph, "demo", .{ .source_name = "raw", .table_name = "missing" }));
302382
}
303383

384+
test "duplicate validation rejects duplicate model and seed unique ids" {
385+
var arena = std.heap.ArenaAllocator.init(std.testing.allocator);
386+
defer arena.deinit();
387+
var graph = Graph{ .allocator = arena.allocator(), .project_name = "demo" };
388+
defer graph.deinit();
389+
390+
try appendNode(&graph, "model", "demo", "model.demo.customers", "customers", true);
391+
try appendNode(&graph, "model", "demo", "model.demo.customers", "customers", true);
392+
try appendNode(&graph, "seed", "demo", "seed.demo.raw_customers", "raw_customers", true);
393+
try appendNode(&graph, "seed", "demo", "seed.demo.raw_customers", "raw_customers", true);
394+
395+
try std.testing.expectError(error.DuplicateModelName, rejectDuplicateModels(&graph));
396+
try std.testing.expectError(error.DuplicateSeedName, rejectDuplicateSeeds(&graph));
397+
}
398+
399+
test "duplicate validation rejects docs exposures macros and macro properties" {
400+
var arena = std.heap.ArenaAllocator.init(std.testing.allocator);
401+
defer arena.deinit();
402+
var graph = Graph{ .allocator = arena.allocator(), .project_name = "demo" };
403+
defer graph.deinit();
404+
405+
try graph.docs.append(graph.allocator, .{ .package_name = "demo", .unique_id = "doc.demo.orders", .name = "orders", .path = "", .original_file_path = "", .block_contents = "" });
406+
try graph.docs.append(graph.allocator, .{ .package_name = "demo", .unique_id = "doc.demo.orders", .name = "orders", .path = "", .original_file_path = "", .block_contents = "" });
407+
try graph.exposures.append(graph.allocator, .{ .package_name = "demo", .unique_id = "exposure.demo.weekly_kpis", .name = "weekly_kpis", .path = "", .original_file_path = "" });
408+
try graph.exposures.append(graph.allocator, .{ .package_name = "demo", .unique_id = "exposure.demo.weekly_kpis", .name = "weekly_kpis", .path = "", .original_file_path = "" });
409+
try appendMacro(&graph, "demo", "format_id");
410+
try appendMacro(&graph, "demo", "format_id");
411+
try graph.macro_properties.append(graph.allocator, .{ .package_name = "demo", .name = "format_id", .patch_path = "macros/schema.yml" });
412+
try graph.macro_properties.append(graph.allocator, .{ .package_name = "demo", .name = "format_id", .patch_path = "macros/schema.yml" });
413+
414+
try std.testing.expectError(error.DuplicateDocName, rejectDuplicateDocs(&graph));
415+
try std.testing.expectError(error.DuplicateExposureName, rejectDuplicateExposures(&graph));
416+
try std.testing.expectError(error.DuplicateMacroName, rejectDuplicateMacros(&graph));
417+
try std.testing.expectError(error.DuplicateMacroProperty, rejectDuplicateMacroProperties(&graph));
418+
}
419+
304420
test "dependency resolution populates sorted unique node and exposure dependencies" {
305421
var arena = std.heap.ArenaAllocator.init(std.testing.allocator);
306422
defer arena.deinit();

0 commit comments

Comments
 (0)