Dynamic seed source configuration & workflow changes#5
Open
mkrahu-z2 wants to merge 12 commits into
Open
Conversation
…tor to use seeds or source depending on the variable setting
…ile that was missing previously
…x_extract on what could be a decimal type field
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe your changes
Adjust the cms_synthetic_connector to leverage either the seed data (default behavior) or source data depending on the status of the
demo_data_onlyvariable indbt_project.ymlHow has this been tested?
Tested locally with duckdb, toggled the
demo_data_onlybetweentrueandfalse, when set totrue, the connector loads the seeds from s3 as expected, when set tofalsethe connector leverages the source locations as defined inmodels/_sources.ymlReviewer focus
Review the implementation and see if this is intuitive for how we can create a 'default' behavior in connectors to allow them t run with synthetic demo data "out of the box" while easily switching to source locations once they have been implemented.
Checklist before requesting a review
(Optional) Gif of how this PR makes you feel
Loom link