Add dbt variable reference documentation to README#1245
Merged
Conversation
Document all user-configurable dbt variables including data source enablement, individual mart toggles, runtime configuration, and infrastructure/schema settings. Fixes #1097
github-project-automation
Bot
moved this from 👀 Ready for Review
to 🚀 Ready to Release
in The Tuva Project
sarahmcmorgan
added a commit
to Chamber-Cardio/tuva
that referenced
this pull request
Mar 17, 2026
* Fix typo: desription → description in intermediate_models.yml (tuva-health#1224) * Add deterministic Tuva dev workflow + outside PR bridge mode (tuva-health#1227) * Add deterministic Tuva dev workflow and outside PR bridge mode * Enforce clean-main branch starts and persistent CI remediation --------- Co-authored-by: Aaron Neiderhiser <aaronneiderhiser@Aarons-MacBook-Air.local> * Add per-database concurrency groups to CI workflows to prevent conflicts (tuva-health#1229) Co-authored-by: Aaron Neiderhiser <anneiderhiser@gmail.com> * [outside-pr 1219] [Bug Fix] add data_source to exclude list for unpivot in data_quality__readmissions (tuva-health#1228) add new field data_source to exclude list for unpivot (#3) Co-authored-by: Sarah Morgan <sarah.morgan@chambercardio.com> * Exclude New Enrollees from disease factor scoring (tuva-health#1223) The CMS NE and SNPNE models are demographic-only regressions with zero disease HCC variables (CMS SAS V2425P1M.TXT lines 207-270). Tuva was incorrectly applying community-model disease HCC coefficients to NE patients, inflating scores by ~1.0 points. Fixes tuva-health#1221 * Implement V28 heart interaction patch zero CC223 when isolated (tuva-health#1222) Implement V28 heart interaction patch — zero CC223 when isolated CC223 (Heart Failure, without CC specification) must be zeroed when no sibling heart CCs (CC221, CC222, CC224, CC225, CC226) are present, per CMS SAS V2825T1M.TXT lines 448-450. Applied before hierarchy suppression. V28 only — V24 is unaffected. Fixes tuva-health#1220 * Cast tuva_last_run to timestamp in core__stg_claims_pharmacy_claim (tuva-health#1230) Regression from PR tuva-health#1191 left tuva_last_run as a bare varchar in pharmacy_claim while eligibility and member_months were already fixed. Fixes tuva-health#1065 * Filter CMS-HCC eligibility staging to Medicare records only (tuva-health#1145) (tuva-health#1233) Coverage month calculation for New vs Continuing enrollment status was counting all payer types (commercial, Medicaid, etc.), inflating months and incorrectly classifying members as Continuing. Filter to payer_type = 'medicare' (with null fallback for backwards compat). * Handle empty pharmacy/medical claim tables in plan overlap test (tuva-health#1231) The check_plan_overlap test incorrectly warns when pharmacy_claim has no data for a data source, since the WHERE clause requires plan presence in both claim tables. Add table-presence guards consistent with the sibling overlap tests. Fixes tuva-health#1113 Co-authored-by: Aaron Neiderhiser <anneiderhiser@gmail.com> * Update cms_hcc__cpt_hcpcs seed file to copy prior year (tuva-health#1218) copy cms_hcc__cpt_hcpcs mappings for next year * Strip decimal points from ICD codes before DQI terminology lookups (tuva-health#1232) DQI validates raw diagnosis/procedure codes against terminology tables that store codes without decimals. Codes with decimals (e.g., J06.9) fail the lookup and are incorrectly flagged as invalid. Apply replace() to strip decimals before joining, matching the existing pattern in claims_preprocessing normalized_input. Affects legacy DQI, DQI atomic checks for institutional/professional diagnosis codes 1-3, procedure codes 1-3, and clinical encounter/ condition/procedure checks (22 join conditions across 13 files). Fixes tuva-health#995 * Include source_code_type in procedure_id to prevent PK violations (tuva-health#1114) (tuva-health#1238) When claim lines have inconsistent procedure_code_type values (e.g., 'icd-10-pcs' on one line and NULL on another), the unpivot produces duplicate rows with different source_code_type but identical procedure_id. Adding source_code_type to the procedure_id construction ensures uniqueness. * Include source_code_type in condition_id to prevent PK violations (tuva-health#1110) (tuva-health#1239) When claim lines have inconsistent diagnosis_code_type values (e.g., 'icd-10-cm' on one line and NULL on another), the intermediate model produces duplicate rows with different source_code_type but identical condition_id. Adding source_code_type to the condition_id construction ensures uniqueness. * Fix OREC/disabled model classification per CMS SAS DISABL/ORIGDS logic (tuva-health#1002) (tuva-health#1240) Make orec age-aware at the source in cms_hcc__int_members: members 65+ with disability OREC now correctly map to the Aged community model (CNA/CFA/CPA) instead of Disabled (CND/CFD/CPD). Add originally_disabled_flag (OREC=1 + age>=65) and thread it through demographic_factors to enrollment_interaction_factors, replacing the scattered age-group hacks across 5 downstream factor models. CMS SAS reference: DISABL = (AGEF < 65 & OREC ne "0") ORIGDS = (OREC = '1') * (DISABL = 0) * Remove ESRD enrollment_status override that excludes OREC=2/3 members (tuva-health#1242) Members with original_reason_entitlement_code '2' (ESRD) or '3' (ESRD + disability) had their enrollment_status overridden to 'ESRD', which doesn't match any demographic factor seed rows, causing these members to be completely excluded from CMS-HCC Part C scoring. OREC indicates why a beneficiary was originally entitled to Medicare, not whether they should receive Part C risk scores. The CMS SAS reference code uses OREC only to compute DISABL and ORIGDS flags, which are already handled correctly by the orec column mapping. Fixes tuva-health#1234 * Relax disease/interaction/hcc-count factor seed joins for institutional model (tuva-health#1243) The CMS-HCC institutional (INS) model uses a single set of coefficients per HCC, without splitting by medicaid_status, dual_status, or orec. The existing joins required exact matches on all dimensions, which incorrectly excluded institutional members whose demographic values didn't match the seed rows. This adds conditional join logic that skips medicaid/dual/orec matching when institutional_status = 'Yes'. * Convert latitude and longitude columns from float to numeric (tuva-health#1241) Replace dbt.type_float() with dbt.type_numeric() for latitude and longitude columns in patient and location staging models. Floating point types introduce rounding artifacts for coordinate values; numeric/decimal types preserve exact precision needed for geospatial accuracy. Fixes tuva-health#794 * Add dbt variable reference documentation to README (tuva-health#1245) Document all user-configurable dbt variables including data source enablement, individual mart toggles, runtime configuration, and infrastructure/schema settings. Fixes tuva-health#1097 * Skip seed population check for non-build dbt commands (tuva-health#1247) The on-run-end hook log_warning_for_seeds() queries seed tables that may not exist yet, causing failures on dbt docs generate, compile, and parse. Guard with flags.WHICH to only run during seed/build/run. Fixes tuva-health#983 * 866 dqi test for diagnosis codes were not unique at the claim header level (tuva-health#1244) * Add DQI check for duplicate diagnosis codes on institutional claims New atomic check that flags institutional claims where the same diagnosis code appears in multiple positions (e.g., diagnosis_code_1 and diagnosis_code_3 both contain the same ICD-10 code). Unpivots all 25 diagnosis code columns per claim and identifies codes with occurrences > 1. Scoped to institutional claims only, where diagnosis codes are expected to be unique at the claim header level. Fixes tuva-health#866 * Add dbt test to validate institutional diagnosis code uniqueness check Verifies that claims flagged as having duplicate diagnosis codes by the DQI atomic check actually contain repeated codes across positions. The test returns any false positives (claims flagged as duplicate that have no actual duplicates). * Fix string concatenation for cross-warehouse compatibility Replace || operator with dbt.concat() in listagg measure to support MS Fabric/SQL Server which does not recognize the || syntax. * 923 align los calculation with cms definition (tuva-health#1246) * Align LOS calculation with CMS definition: datediff + 1 (tuva-health#923) CMS defines length of stay as discharge_date - admit_date + 1, where the admission day counts as day 1. A same-day admit/discharge = LOS 1. Previously datediff alone returned 0 for same-day stays. Updated across all 9 encounter grain models and readmissions. * Fix LOS calculation to use max(1, datediff) instead of datediff + 1 Per CMS definition, the day of admission counts but not the day of discharge, with same-day cases counting as 1 day. The datediff already gives the correct result for multi-day stays; only same-day (datediff=0) needs the floor to 1. The previous datediff + 1 overcounted by 1 for every multi-day stay. Applied across all 9 encounter grain models and the readmissions encounter_augmented model. * Preserve file_date as timestamp instead of casting to date (tuva-health#1249) The core staging models for medical_claim and pharmacy_claim were using try_to_cast_date which truncated time information. Changed to cast as timestamp to match the eligibility staging model and preserve any time component from the source data. Updated core_models.yml metadata to reflect the timestamp data type. * Add ICD-9-CM normalization to core conditions staging model (tuva-health#1252) Previously only ICD-10-CM codes were normalized via terminology lookup, leaving ICD-9-CM conditions with NULL normalized_code_type, normalized_code, and normalized_description. Added left join to terminology__icd_9_cm and COALESCE logic so both code systems are properly normalized. Closes tuva-health#1068 * Defer monthly collection date expansion in eligible conditions (tuva-health#1253) The monthly collection dates join was applied in each claim-type CTE (professional, inpatient, outpatient), expanding every claim by up to 12x before the condition join. This created unnecessarily large intermediate datasets. Now claims join to yearly-aggregated collection dates (1 row per year) for initial eligibility filtering, and the monthly expansion is deferred to after the condition join where the dataset is much smaller. The output schema is unchanged so no downstream models are affected. Closes tuva-health#1122 * Accept CDT dental codes (Dxxxx) in DQI HCPCS validation (tuva-health#1248) * Accept CDT dental codes (Dxxxx) in DQI HCPCS validation CDT codes follow the pattern D followed by 4 characters and are valid procedure codes that cannot be added to the terminology table due to copyright. Recognize them as valid via pattern match so they are not flagged as invalid HCPCS codes. * Fix Fabric compatibility: use length macro instead of raw length() Replace raw length() calls with the cross-database the_tuva_project.length() macro which dispatches to len() on Fabric. This fixes the CI failure: 'length' is not a recognized built-in function name on SQL Server/Fabric. * Add SNPNE (SNP New Enrollee) scoring model support (tuva-health#1254) * Add SNPNE (SNP New Enrollee) scoring model support Propagate institutional_snp_flag from core eligibility through the CMS-HCC pipeline (stg_eligibility → int_members → int_demographic_factors). SNP New Enrollees are now scored with SNPNE-specific coefficients (risk_model_code = 'SNPNE') instead of regular NE coefficients. The seed demographic_factors table needs 'SNP New' enrollment_status rows populated with SNPNE coefficients from the CMS rate announcement. SNPNE remains excluded from disease factors (demographic-only model). Closes tuva-health#1237 * Use existing plan_segment='C-SNP' seed data for SNPNE coefficients The demographic factors seed already contains SNPNE coefficients stored with plan_segment='C-SNP'. Use a separate seed_snpne_factors CTE to query these rows instead of requiring a new enrollment_status value. This eliminates the need for any cloud seed data changes. * Add reason column to hcc_recapture__int_all_hccs uniqueness test (tuva-health#1258) The unique_combination_of_columns test was missing the reason column, causing false-positive duplicate failures when rows differed only by reason value. * Add 3-mode PR CI with Snowflake-small default and /ci mode commands (tuva-health#1260) * Add 3-mode PR CI with Snowflake-small default and /ci mode commands * Improve CI run and check names with mode and warehouse --------- Co-authored-by: Aaron Neiderhiser <aaronneiderhiser@Aarons-MacBook-Air.local> * Add ingest_datetime to core.medical_claim and core.pharmacy_claim (tuva-health#1263) Pulls ingest_datetime through from normalized input to core staging and final models for medical_claim and pharmacy_claim. This field was already present in core.eligibility. Having both file_date and ingest_datetime in core strengthens traceability between raw data, input layer, and core, particularly when a payer sends the same claim across multiple files. * Fix OREC=2 (ESRD) members under 65 incorrectly assigned Aged model (tuva-health#1265) Per CMS SAS: DISABL = (AGEF < 65 & OREC ne "0"). Any OREC other than '0' combined with age < 65 should select the Disabled model. OREC=2 (ESRD) was incorrectly grouped with OREC=0 as 'Aged', causing under-65 ESRD members to receive CNA/CFA/CPA instead of CND/CFD/CPD. The age>=65 override in int_demographic_factors already handles the transition to Aged for older members, so the base OREC column must reflect the pre-override disability logic. * Gate disabled interaction factors by DISABL flag (tuva-health#1268) Institutional members with OREC=0 (Old Age) and age < 65 were incorrectly receiving disabled interaction coefficients (e.g. INS_DISABLED_HF_CHF, INS_DISABLED_PRESSURE_ULCER). Per CMS SAS: DISABL = (AGEF < 65 & OREC ne "0") — these factors should only apply when the member is both under 65 and has a non-aged OREC. Added orec and age_group to the demographics CTE and filtered the interaction join to exclude members where DISABL=0. * Fix duplicate CI runs by dispatching mode changes (tuva-health#1266) * Fix CI mode command to dispatch single PR run * Dispatch CI mode runs on PR head ref * Pin DuckDB to 1.4.4 for MotherDuck CI --------- Co-authored-by: Aaron Neiderhiser <aaronneiderhiser@Aarons-MacBook-Air.local> * Bump dbt package version to 0.17.1 (tuva-health#1273) Co-authored-by: Aaron Neiderhiser <aaronneiderhiser@Aarons-MacBook-Air.local> --------- Co-authored-by: Chase Jones <60072797+chase-jones@users.noreply.github.com> Co-authored-by: Aaron Neiderhiser <anneiderhiser@gmail.com> Co-authored-by: Aaron Neiderhiser <aaronneiderhiser@Aarons-MacBook-Air.local> Co-authored-by: andreas-homeward <andreas.martinson@homewardhealth.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Document all user-configurable dbt variables including data source enablement, individual mart toggles, runtime configuration, and infrastructure/schema settings.
Fixes #1097