Skip to content

Commit 369bc4b

Browse files
authored
Extract parser scalar helpers (#37)
1 parent 94a186a commit 369bc4b

3 files changed

Lines changed: 117 additions & 68 deletions

File tree

PLAN.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -569,7 +569,7 @@ Exit criteria:
569569

570570
- M0 is complete as the Zig `0.16.0` runtime scaffold.
571571
- M1 has started on stacked branches with native Zig artifact-first parser slices.
572-
- M1A has started with behavior-preserving `src/project/types.zig`, `src/project/selector.zig`, `src/project/manifest.zig`, `src/project/util.zig`, `src/project/config.zig`, `src/project/fs.zig`, `src/project/jinja.zig`, and `src/project/resolve.zig` extractions. `src/project/manifest.zig` owns selected-resource JSON and partial `manifest.json` writing with native tests for selected JSON shape, JSON escaping, exposure dependency ordering, disabled-resource filtering, and graph-map output. `src/project/util.zig` owns shared display, membership, append-dedup, string sorting, and narrow YAML scalar/list helpers. `src/project/config.zig` owns `dbt_project.yml` loading and project path/docs config parsing. `src/project/fs.zig` owns deterministic resource file discovery and Linux directory traversal helpers. `src/project/jinja.zig` owns lexical Jinja call, parenthesis, quoted string, and literal argument helpers while graph-aware scanning remains in `src/project.zig`. `src/project/resolve.zig` owns graph lookup/count helpers, low-level ref/source resolution helpers, and dependency-map mutation for refs, sources, and known macro dependencies. `src/project.zig` remains the public parser/list facade and still owns most loader, resource parser, scanner, and resolver orchestration code until follow-up extractions move those pieces behind focused internal modules.
572+
- M1A has started with behavior-preserving `src/project/types.zig`, `src/project/selector.zig`, `src/project/manifest.zig`, `src/project/util.zig`, `src/project/config.zig`, `src/project/fs.zig`, `src/project/jinja.zig`, `src/project/resolve.zig`, and `src/project/parse.zig` extractions. `src/project/manifest.zig` owns selected-resource JSON and partial `manifest.json` writing with native tests for selected JSON shape, JSON escaping, exposure dependency ordering, disabled-resource filtering, and graph-map output. `src/project/util.zig` owns shared display, membership, append-dedup, string sorting, and narrow YAML scalar/list helpers. `src/project/config.zig` owns `dbt_project.yml` loading and project path/docs config parsing. `src/project/fs.zig` owns deterministic resource file discovery and Linux directory traversal helpers. `src/project/jinja.zig` owns lexical Jinja call, parenthesis, quoted string, and literal argument helpers while graph-aware scanning remains in `src/project.zig`. `src/project/resolve.zig` owns graph lookup/count helpers, low-level ref/source resolution helpers, and dependency-map mutation for refs, sources, and known macro dependencies. `src/project/parse.zig` owns narrow parser scalar helpers for YAML booleans, JSON-compatible scalar classification, and generic-test YAML item names with native tests. `src/project.zig` remains the public parser/list facade and still owns most loader, resource parser, scanner, and resolver orchestration code until follow-up extractions move those pieces behind focused internal modules.
573573
- `dxt parse` now targets the supported Tier 0 subset: project name/model paths/seed paths/macro paths, project and package model path configs for literal `+materialized`, `+tags`, and model/seed `+docs.node_color`, root-project model config overrides for installed packages, SQL model discovery, CSV seed discovery, installed package SQL model and CSV seed discovery from `dbt_packages`, source discovery, installed package source discovery, exposure discovery, installed package exposure discovery, project macro discovery, installed package macro discovery from `dbt_packages`, macro property YAML for project macro descriptions and arguments, project and package docs block discovery, literal `ref` to models or seeds, two-argument package refs, package-local refs in installed package models and exposures, unique installed-package fallback for unqualified refs, literal `source`, package-local sources in installed package models, unique installed-package fallback for unqualified sources, literal `doc` in project and package descriptions, inline `config(materialized=..., tags=...)`, known project/package-qualified/package-local macro call dependencies, narrow project and package YAML model properties for scalar descriptions, simple columns, tags, materialization, disabled SQL models, dbt-shaped `unique`, `not_null`, `accepted_values`, and `relationships` generic test nodes, model/test `refs` and `sources` artifact fields, dependency maps, and deterministic partial `manifest.json`. The manifest includes the v12 top-level maps needed by the M1 artifact shape and is covered by a pinned local dbt Manifest v12 schema slice. YAML generic test arguments are currently supported for scalar values plus inline and block lists required by public Jaffle Shop DuckDB-style tests.
574574
- `dxt ls` now lists dbt-selectable resources from the same parser graph with stable text/JSON output and basic name/FQN wildcards, tag wildcards, slash-aware `path:` wildcards, exact `package:`/`package:this`, `source:` wildcards including package-qualified source selectors, `exposure:` wildcards, `resource_type:`, `test_type:generic`, config materialization, comma intersection, whitespace union, multi-argument selector lists, repeated selector flags, leading/trailing `+` graph expansion, and exact exclude filters; macros are emitted in artifacts but not exposed as `ls` resources.
575575
- Synthetic fixtures cover one model, model refs, seed refs, source refs, exposure refs to models and sources, combined source/model YAML, inline config/tag selection, config materialization selection, comma-intersection selection, YAML model properties and columns, emitted `unique`, `not_null`, `accepted_values`, and `relationships` generic test nodes, project macro artifacts and macro properties, configured `macro-paths` replacing the default macro directory, installed package macros with package-qualified calls and package-local macro calls, installed package models, seeds, sources, docs, exposures, package YAML model properties, root package config overrides, and package-qualified/package-local refs/sources, macro calls recorded in model and macro `depends_on.macros`, docs blocks with literal `doc` descriptions, disabled models, disabled ref diagnostics, unmatched model-property warnings, duplicate model and docs diagnostics, unsupported dynamic ref/doc diagnostics, missing doc diagnostics, malformed docs block diagnostics, unresolved package macro diagnostics, and unsupported unknown macro-call diagnostics.

src/project.zig

Lines changed: 4 additions & 67 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ const Io = std.Io;
33
const project_config = @import("project/config.zig");
44
const project_fs = @import("project/fs.zig");
55
const project_jinja = @import("project/jinja.zig");
6+
const project_parse = @import("project/parse.zig");
67
const project_resolve = @import("project/resolve.zig");
78
const manifest = @import("project/manifest.zig");
89
const selector = @import("project/selector.zig");
@@ -45,6 +46,9 @@ const splitKeyValue = util.splitKeyValue;
4546
const parseInlineStringList = util.parseInlineStringList;
4647
const dupTrimmedScalar = util.dupTrimmedScalar;
4748
const sortStrings = util.sortStrings;
49+
const parseBool = project_parse.parseBool;
50+
const parseJsonScalar = project_parse.parseJsonScalar;
51+
const testNameFromYamlItem = project_parse.testNameFromYamlItem;
4852
const findKeyword = project_jinja.findKeyword;
4953
const findMatchingParen = project_jinja.findMatchingParen;
5054
const findValueStart = project_jinja.findValueStart;
@@ -1551,15 +1555,6 @@ fn appendMetaEntry(allocator: std.mem.Allocator, entries: *std.ArrayList(MetaEnt
15511555
sortMetaEntries(entries.items);
15521556
}
15531557

1554-
fn parseJsonScalar(allocator: std.mem.Allocator, value: []const u8) !JsonScalar {
1555-
const trimmed = std.mem.trim(u8, value, " \t\r");
1556-
const unquoted = try dupTrimmedScalar(allocator, trimmed);
1557-
if (std.mem.eql(u8, trimmed, "true") or std.mem.eql(u8, trimmed, "false")) return .{ .text = unquoted, .kind = .bool };
1558-
if (std.mem.eql(u8, trimmed, "null")) return .{ .text = unquoted, .kind = .null };
1559-
if (isJsonNumber(trimmed)) return .{ .text = unquoted, .kind = .number };
1560-
return .{ .text = unquoted, .kind = .string };
1561-
}
1562-
15631558
fn currentGenericTestDef(graph: *Graph, model_index: usize, current_column: ?usize, target: TestTarget, test_index: usize) !*GenericTestDef {
15641559
if (target == .model) return &graph.model_properties.items[model_index].tests.items[test_index];
15651560
if (target == .column) {
@@ -1840,64 +1835,6 @@ fn packageNameFromMacroUniqueId(unique_id: []const u8) ?[]const u8 {
18401835
return unique_id[package_start..package_end];
18411836
}
18421837

1843-
fn parseBool(value: []const u8) !bool {
1844-
const trimmed = std.mem.trim(u8, value, " \t\r");
1845-
if (std.ascii.eqlIgnoreCase(trimmed, "true")) return true;
1846-
if (std.ascii.eqlIgnoreCase(trimmed, "false")) return false;
1847-
return error.UnsupportedYaml;
1848-
}
1849-
1850-
fn isJsonNumber(value: []const u8) bool {
1851-
if (value.len == 0) return false;
1852-
var i: usize = 0;
1853-
if (value[i] == '-') {
1854-
i += 1;
1855-
if (i == value.len) return false;
1856-
}
1857-
if (value[i] == '0') {
1858-
i += 1;
1859-
if (i < value.len and std.ascii.isDigit(value[i])) return false;
1860-
} else if (value[i] >= '1' and value[i] <= '9') {
1861-
i += 1;
1862-
while (i < value.len and std.ascii.isDigit(value[i])) : (i += 1) {}
1863-
} else {
1864-
return false;
1865-
}
1866-
if (i < value.len and value[i] == '.') {
1867-
i += 1;
1868-
var frac_digits: usize = 0;
1869-
while (i < value.len and std.ascii.isDigit(value[i])) : (i += 1) {
1870-
frac_digits += 1;
1871-
}
1872-
if (frac_digits == 0) return false;
1873-
}
1874-
if (i < value.len and (value[i] == 'e' or value[i] == 'E')) {
1875-
i += 1;
1876-
if (i < value.len and (value[i] == '+' or value[i] == '-')) i += 1;
1877-
var exp_digits: usize = 0;
1878-
while (i < value.len and std.ascii.isDigit(value[i])) : (i += 1) {
1879-
exp_digits += 1;
1880-
}
1881-
if (exp_digits == 0) return false;
1882-
}
1883-
return i == value.len;
1884-
}
1885-
1886-
test "json number parser rejects invalid leading zero forms" {
1887-
try std.testing.expect(isJsonNumber("0"));
1888-
try std.testing.expect(isJsonNumber("-12.5e+3"));
1889-
try std.testing.expect(!isJsonNumber("007"));
1890-
try std.testing.expect(!isJsonNumber("-01"));
1891-
try std.testing.expect(!isJsonNumber("1."));
1892-
}
1893-
1894-
fn testNameFromYamlItem(allocator: std.mem.Allocator, value: []const u8) ![]const u8 {
1895-
const trimmed = std.mem.trim(u8, value, " \t\r");
1896-
if (trimmed.len == 0) return error.UnsupportedYaml;
1897-
const colon = std.mem.indexOfScalar(u8, trimmed, ':') orelse trimmed.len;
1898-
return try dupTrimmedScalar(allocator, trimmed[0..colon]);
1899-
}
1900-
19011838
test "sql scanner extracts refs sources and config tags from jinja spans" {
19021839
var arena = std.heap.ArenaAllocator.init(std.testing.allocator);
19031840
defer arena.deinit();

src/project/parse.zig

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
const std = @import("std");
2+
const types = @import("types.zig");
3+
const util = @import("util.zig");
4+
5+
const JsonScalar = types.JsonScalar;
6+
const dupTrimmedScalar = util.dupTrimmedScalar;
7+
8+
pub fn parseBool(value: []const u8) !bool {
9+
const trimmed = std.mem.trim(u8, value, " \t\r");
10+
if (std.ascii.eqlIgnoreCase(trimmed, "true")) return true;
11+
if (std.ascii.eqlIgnoreCase(trimmed, "false")) return false;
12+
return error.UnsupportedYaml;
13+
}
14+
15+
pub fn parseJsonScalar(allocator: std.mem.Allocator, value: []const u8) !JsonScalar {
16+
const trimmed = std.mem.trim(u8, value, " \t\r");
17+
const unquoted = try dupTrimmedScalar(allocator, trimmed);
18+
if (std.mem.eql(u8, trimmed, "true") or std.mem.eql(u8, trimmed, "false")) return .{ .text = unquoted, .kind = .bool };
19+
if (std.mem.eql(u8, trimmed, "null")) return .{ .text = unquoted, .kind = .null };
20+
if (isJsonNumber(trimmed)) return .{ .text = unquoted, .kind = .number };
21+
return .{ .text = unquoted, .kind = .string };
22+
}
23+
24+
fn isJsonNumber(value: []const u8) bool {
25+
if (value.len == 0) return false;
26+
var i: usize = 0;
27+
if (value[i] == '-') {
28+
i += 1;
29+
if (i == value.len) return false;
30+
}
31+
if (value[i] == '0') {
32+
i += 1;
33+
if (i < value.len and std.ascii.isDigit(value[i])) return false;
34+
} else if (value[i] >= '1' and value[i] <= '9') {
35+
i += 1;
36+
while (i < value.len and std.ascii.isDigit(value[i])) : (i += 1) {}
37+
} else {
38+
return false;
39+
}
40+
if (i < value.len and value[i] == '.') {
41+
i += 1;
42+
var frac_digits: usize = 0;
43+
while (i < value.len and std.ascii.isDigit(value[i])) : (i += 1) {
44+
frac_digits += 1;
45+
}
46+
if (frac_digits == 0) return false;
47+
}
48+
if (i < value.len and (value[i] == 'e' or value[i] == 'E')) {
49+
i += 1;
50+
if (i < value.len and (value[i] == '+' or value[i] == '-')) i += 1;
51+
var exp_digits: usize = 0;
52+
while (i < value.len and std.ascii.isDigit(value[i])) : (i += 1) {
53+
exp_digits += 1;
54+
}
55+
if (exp_digits == 0) return false;
56+
}
57+
return i == value.len;
58+
}
59+
60+
pub fn testNameFromYamlItem(allocator: std.mem.Allocator, value: []const u8) ![]const u8 {
61+
const trimmed = std.mem.trim(u8, value, " \t\r");
62+
if (trimmed.len == 0) return error.UnsupportedYaml;
63+
const colon = std.mem.indexOfScalar(u8, trimmed, ':') orelse trimmed.len;
64+
return try dupTrimmedScalar(allocator, trimmed[0..colon]);
65+
}
66+
67+
test "parseBool accepts YAML bool scalars case-insensitively" {
68+
try std.testing.expect(try parseBool(" true "));
69+
try std.testing.expect(try parseBool("FALSE\r") == false);
70+
try std.testing.expectError(error.UnsupportedYaml, parseBool("yes"));
71+
}
72+
73+
test "parseJsonScalar classifies JSON-compatible YAML scalars" {
74+
var arena = std.heap.ArenaAllocator.init(std.testing.allocator);
75+
defer arena.deinit();
76+
const allocator = arena.allocator();
77+
78+
const string_scalar = try parseJsonScalar(allocator, "\"blue\"");
79+
try std.testing.expectEqual(.string, string_scalar.kind);
80+
try std.testing.expectEqualStrings("blue", string_scalar.text);
81+
82+
const number_scalar = try parseJsonScalar(allocator, "-12.5e+3");
83+
try std.testing.expectEqual(.number, number_scalar.kind);
84+
try std.testing.expectEqualStrings("-12.5e+3", number_scalar.text);
85+
86+
const bool_scalar = try parseJsonScalar(allocator, "false");
87+
try std.testing.expectEqual(.bool, bool_scalar.kind);
88+
try std.testing.expectEqualStrings("false", bool_scalar.text);
89+
90+
const null_scalar = try parseJsonScalar(allocator, "null");
91+
try std.testing.expectEqual(.null, null_scalar.kind);
92+
try std.testing.expectEqualStrings("null", null_scalar.text);
93+
}
94+
95+
test "json number parser rejects invalid leading zero forms" {
96+
try std.testing.expect(isJsonNumber("0"));
97+
try std.testing.expect(isJsonNumber("-12.5e+3"));
98+
try std.testing.expect(!isJsonNumber("007"));
99+
try std.testing.expect(!isJsonNumber("-01"));
100+
try std.testing.expect(!isJsonNumber("1."));
101+
}
102+
103+
test "testNameFromYamlItem reads scalar and mapping test names" {
104+
var arena = std.heap.ArenaAllocator.init(std.testing.allocator);
105+
defer arena.deinit();
106+
const allocator = arena.allocator();
107+
108+
try std.testing.expectEqualStrings("not_null", try testNameFromYamlItem(allocator, " not_null "));
109+
try std.testing.expectEqualStrings("relationships", try testNameFromYamlItem(allocator, "relationships:"));
110+
try std.testing.expectEqualStrings("accepted_values", try testNameFromYamlItem(allocator, "\"accepted_values\": {values: [a, b]}"));
111+
try std.testing.expectError(error.UnsupportedYaml, testNameFromYamlItem(allocator, " "));
112+
}

0 commit comments

Comments
 (0)