refactor(integration_tests): bootstrap pipeline from Tuva package seeds#1344
Open
rabee05 wants to merge 28 commits into
Open
refactor(integration_tests): bootstrap pipeline from Tuva package seeds#1344rabee05 wants to merge 28 commits into
rabee05 wants to merge 28 commits into
Conversation
…_input - Add source_input source: resolves to the synthetic seed schema when use_synthetic_data is true, else the configured input_database/input_schema. - Input models select the input-layer columns from source_input. - DRY the input_layer schema config with a YAML anchor.
- Add header stubs for the condition, encounter, location, medication, practitioner, and procedure synthetic seeds. - Enable synthetic seeds only when use_synthetic_data is true, defaulting to false so an unset var no longer errors. - Use YAML anchors for the shared column type definitions.
Provide an athena__ dispatch built on Trino json_object/json_format/array_agg (Athena engine v3). Marked as needing Athena verification.
Return the last element of a delimited string. split_part(..., -1) works on most adapters but Athena/Trino rejects negative indexes, so provide athena__ and fabric__ implementations alongside the default.
Wrap the load_seed cast so empty strings, not just the null marker, are converted to NULL.
Trino/Athena has no two-argument ltrim(string, chars); strip the leading character set with regexp_replace instead.
BigQuery rejects doubled single quotes ('') and reads them as adjacent string
literals, breaking metric names with apostrophes such as "Parkinson's". Escape
with backslash on BigQuery; keep quote-doubling for the other adapters.
BigQuery requires UNION ALL or UNION DISTINCT; emit "union distinct" on BigQuery and keep the bare "union" for the other adapters.
The empty-seed on-run-end check ran one COUNT(*) per seed serially, adding minutes of round-trip latency on large seed sets. Combine all counts into a single UNION ALL query (cross-database safe) and report from the result set.
Trino does not implicitly coerce string literals to dates when comparing against date/timestamp columns, so wrap date-literal bounds in date(...) on Athena. Numeric bounds are left unchanged.
…ge_is_100 Drop Athena from the STRING-cast branch so it falls through to the varchar() default, matching Trino's preferred string type.
Replace the non-portable month() call (which BigQuery does not support) with the cross-database date_part macro so the test runs on all adapters.
…de_flag The table and its column share the name icd_10_cm, so BigQuery resolved the unqualified reference to the whole-row struct, breaking the UNION ALL with a type mismatch. Alias the table and qualify the columns.
dbt 1.11 deprecates custom keys placed directly on column objects (CustomKeyInObjectDeprecation). Nest required_for_data_marts under the column's config.meta across the input-layer models; the data-dictionary consumer already reads it from config.meta first, so behavior is unchanged.
Replace the inline fabric/default split-part branches for diagnosis_rank with the last_split_part macro, which also covers Athena.
BigQuery types a bare null as INT64, which clashed with the STRING claim_id from int_coded_hccs in the downstream UNION ALL. Cast the null to a string.
dbt 1.11 deprecates top-level generic-test arguments (MissingArgumentsPropertyInGenericTestDeprecation). Move the accepted_values "values" under an arguments block for the hcc_status columns.
Athena/Trino exhausts query resources computing dense_rank() over the full 300M+ row union because of the global sort. On Athena, rank the distinct dimension combinations and join them back (NULL-safe via is not distinct from). Other adapters keep the original single-pass window; summary_sk is unchanged.
Cast the tuva_last_run literal to a timestamp and alias it so both UNION ALL branches share an explicit type instead of an untyped string literal.
Replace native NUMERIC casts with the dbt.type_numeric macro so the chart value column is portable across adapters.
Type rbcs_famnumb and current_flag as decimal(38,0) on Athena to match the numeric type used by the other adapters, instead of int.
Add commented input_database/input_schema examples next to the synthetic data config to show how to point the integration tests at a real input layer.
Input models read via source(), which creates no dependency on the synthetic_data__* seeds, so they raced ahead of the seeds and failed with "table not found". Add a synthetic-mode-only depends_on comment referencing the matching seed to force seed-before-model ordering, keeping the native source() in the FROM clause.
Replace repeated inline cross-database type expressions in the concept_library, provider_data, reference_data, and terminology seed configs with shared YAML anchors, cutting duplication.
✅ Deploy Preview for thetuvaproject canceled.
|
There was a problem hiding this comment.
Pull request overview
This PR updates the integration test / seed + test utilities to improve cross-database compatibility (notably Athena/Trino) while refactoring seed configuration to reduce repetition via YAML anchors.
Changes:
- Improved portability of tests by using cross-db macros (
date_part) and qualifying column references in terminology tests. - Adjusted generic test SQL rendering to avoid Athena-incompatible type casts and added Athena-specific handling for date bounds in
expect_column_values_to_be_between. - Refactored terminology seed configs to use shared YAML anchors for common column type definitions.
Reviewed changes
Copilot reviewed 60 out of 60 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/test_terminology_icd_10_cm_billable_code_flag.sql | Qualifies selected columns via table alias to avoid ambiguity and improve robustness. |
| tests/test_recapture_rates.sql | Replaces month() with cross-db date_part macro for portability. |
| tests/generic/test_warn_if_null_percentage_is_100.sql | Adjusts adapter-specific casting so Athena uses varchar() rather than STRING. |
| tests/generic/dbt_expectations_tests.sql | Adds an Athena/Trino-specific branch to correctly handle date literal bounds in “between” tests. |
| seeds/terminology/terminology_seeds.yml | Introduces shared YAML anchors to deduplicate cross-db seed column type definitions. |
| macros/cross_database_utils/last_split_part.sql | Provides a cross-db “last split part” helper; Athena implementation needs a corrected index calculation for multi-character delimiters. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
The athena split_part index used the number of characters removed by replace(), which only equals the delimiter count for single-character delimiters. For multi-char delimiters it overshoots: "a||b||c" with "||" gave index 5 -> '' instead of 3 -> "c". Divide by the delimiter length and cast to integer so it resolves to the correct last part. The only current use passes a single-char '_' delimiter, so its output is unchanged.
…ider_attribution Commit 37c7721 accidentally dropped these two metadata columns, so input_layer__provider_attribution's select * stopped matching its disabled-branch schema. Restore the guarded exprs that emit cast(null) when the source lacks the columns (as the synthetic seed does), keeping output consistent with the other input models.
…ution Came from the synthetic seed CSV, not the input_layer schema (which uses member_id). Unused downstream; removing it keeps synthetic mode working and avoids a crash in real-data mode where the column may be absent.
…ation Pre-existing gap unrelated to this PR: person_id was missing the meta.data_type that every other column carries. Fixed while here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1332
Summary
This updates the integration test project so it can build the entire Tuva pipeline end to end using only the synthetic seed data shipped inside the Tuva package. There is no source data to stage and nothing to load by hand. The host project can run on the package seeds out of the box, or override them with its own data. It also fixes a batch of cross-database and data-quality bugs found along the way.
What changed
synthetic_data__*seeds, gated onuse_synthetic_data. Seed-before-model ordering is enforced so a singledbt buildalways works, with no more "table not found" races.input_database/input_schemaat its own input layer instead, running the same pipeline on real data.create_json_object,ltrim, andlast_split_partimplementations, plus fixes for empty-string seed loads, date bounds, varchar casts, and decimal seed types.union distinct,tuva_last_runcast, anddbt.type_numeric.config.meta, testarguments:).How to test
Two modes, depending on what you are doing:
use_synthetic_data: true).dbt build --full-refreshbuilds the whole pipeline from the package's synthetic seeds, no setup required. Best for trying Tuva out end to end.use_synthetic_data: false). Pointinput_database/input_schemaat any database or schema you want, thendbt build --full-refresh --vars '{use_synthetic_data: false, input_database: <db>, input_schema: <schema>}'. Use this to build against real data or to test new features and scenarios the synthetic seeds do not cover.Breaking changes
None. The Tuva package contract is unchanged.
Author: SnowQuery Healthcare Data Engineering & Architecture Consulting