fix(grz-cli): Submission id upload tracking#556
Merged
Conversation
virag-compbio
changed the title
virag-compbio
changed the title
tedil
reviewed
May 4, 2026
virag-compbio
force-pushed
the
fix/submission-id-upload-tracking
branch
from
d1a2126 to
5193ec8
Compare
tedil
reviewed
May 5, 2026
tedil
left a comment
tedil
left a comment
Collaborator
There was a problem hiding this comment.
Now that we have submission_id in the base State class, should we include the submission_id (and the checks for that) in all of the progress logs for consistent behaviour?
virag-compbio
force-pushed
the
fix/submission-id-upload-tracking
branch
from
c79c964 to
2eab697
Compare
Include submission_id in UploadState when setting upload success/failure Add submission_id mismatch check to determine if reupload is needed Improve log message to include submission_id for already uploaded files
…progress tracking
…grz-check refactor
virag-compbio
force-pushed
the
fix/submission-id-upload-tracking
branch
from
2eab697 to
8cfe3f1
Compare
tedil
requested changes
May 7, 2026
tedil
left a comment
tedil
left a comment
Collaborator
There was a problem hiding this comment.
I think this is missing some submission_id writing/reading in some of the states/checks, please ensure the submission_id is passed to every ValidationState, DecryptionState etc., and that it is also checked for during the respective checks / progress updates.
I also recommend having a test that checks that e.g., validation on an "unseeded" file works (logger.get_state(...).get("submission_id") == expected_id), the current manual "seeding" (aka injection) seems a bit odd to me
added 2 commits
May 8, 2026 11:15
virag-compbio
force-pushed
the
fix/submission-id-upload-tracking
branch
from
67a7e1c to
db72e71
Compare
virag-compbio
added a commit
that referenced
this pull request
May 11, 2026
Previously, the upload worker would skip uploading a file if there was a cached 'upload_successful' state, even if the cached state was from a completely different submission ID. This change adds the 'submission_id' to the cache, ensuring that files are properly re-uploaded for new submissions. --------- Co-authored-by: Virag Sharma <viragsharma@w620-mb006-wl.local> Co-authored-by: Virag Sharma <viragsharma@w620-mb006-wl.speedport.ip>
github-merge-queue Bot
pushed a commit
that referenced
this pull request
May 12, 2026
## grz-tools release (2026-05-11) This release updates multiple packages in the `grz-tools` monorepo. See the full changelog and per-package details linked below. ### Versions - `grzctl` `2.0.0` (from `1.4.0`) - `grz-db` `2.0.0` (from `1.2.0`) - `grz-common` `2.0.0` (from `1.6.1`) - `grz-cli` `1.6.0` (from `1.5.1`) - `grz-pydantic-models` `2.6.0` (from `2.5.0`) - `grz-check` `0.3.0` (from `0.2.1`) --- ##⚠️ Breaking / behavior changes - `grzctl` now automatically updates submission state in the DB on subcommand invocation (default `--update-db`; use `--no-update-db` to disable). The DB state enum adds `REPORTING`. (#501) - `grzctl pruefbericht submit` now requires `--submission-id` so the submission can be updated consistently in the DB. (#525) - `grz-cli` enforces a version policy via S3 `version.json` for `upload` / `submit` operations. (#477) - Added `selected_for_qc` column and QC workflow version recording; tools now require/record QC workflow version and log `grzctl` runtime version. (#531, #561) - `grzctl db submission populate-qc` now requires `--qc-workflow-version` (or `GRZCTL_QC_WORKFLOW_VERSION`) to be provided so the QC workflow version can be recorded in the DB. (#561) - `grzctl` introduces `backfill` to re-read metadata from archives and other state/reporting improvements (impacting automation that reads/writes the submission DB). (#559) - Dropped the Python dependency `pysam` from `grz-cli`, `grzctl`, and `grz-common` (installs are lighter; environments that relied on it being installed transitively must install it explicitly). (#565) --- ## Highlights ### grz-cli (`1.6.0`) - Version policy checks for `upload` and `submit` via S3 `version.json` (#477) - Multiple config files are merged into one effective configuration (#490) - `upload` now prints the generated submission ID to stdout (can affect scripts that parse command output). (#556) ### grzctl (`2.0.0`) - `db list`: state-based filtering, with “latest” view by default (#523) - `db submission show`: `--json` output (#510) - Prüfbericht generation using submission ID from database (#525) - `report`: makes `identifiers.le` optional (#488) - `report` supports `--with-submission-ids`. (#539) - Improved redaction during archival. (#517) - Add `selected_for_qc` column to the submission database (#531) - `populate-qc`: accept `THRESHOLD NOT MET` as valid QC status (#529) - Fix displayed QC stats (#508) - Introduce `backfill` to re-read metadata from the archive (#559) - Require and record QC workflow version; log `grzctl` runtime versions (#561) - `db tui`: add `--quarter`/`--year` selection for the “Detailed QC by LE” overview. (#541) ### grz-pydantic-models (`2.6.0`) - Enforce differing checksums for sequencing data (#496) - Fix MII consent period to use datetime instead of date (#491) - Change `minReadLength` default from `0` to `-1` (#506) ### grz-check (`0.3.0`) - Python bindings now power checksum validation across the stack (#554) - Checksum calculation is now exposed correctly in the Python API (#5dce167) - `grz-check` is now built and published as a Python package via `pyproject.toml`/maturin. (#543) ### Maintenance - Dependency updates (#498) - Python crypto dependency updates: `pynacl` and `crypt4gh` --- ## Rollout notes (ops) - Apply the submission DB migrations before using this release. The schema changes include `selected_for_qc`, QC workflow version logging, `grzctl` runtime version logging, and the new `REPORTING` state. - Submission DB migrations in this release also include the `qc_queue` table and additional `submissions` fields (`submission_size`, `submission_metadata`)—make sure your DB is migrated to `head` before running the updated tools. - Update any automation that calls `grzctl pruefbericht submit` to pass `--submission-id`. - If you enforce LE uploads via S3, deploy the `version.json` policy for `grz-cli`. - Deploy the matching `grz-check` Python bindings where Python-based checksum validation is used. --- ## Full details - Full changelog: [changes.md](changes.md) - Per-package changelogs: - [packages/grz-cli/CHANGELOG.md](packages/grz-cli/CHANGELOG.md) - [packages/grzctl/CHANGELOG.md](packages/grzctl/CHANGELOG.md) - [packages/grz-common/CHANGELOG.md](packages/grz-common/CHANGELOG.md) - [packages/grz-db/CHANGELOG.md](packages/grz-db/CHANGELOG.md) - [packages/grz-pydantic-models/CHANGELOG.md](packages/grz-pydantic-models/CHANGELOG.md) - [packages/grz-check/CHANGELOG.md](packages/grz-check/CHANGELOG.md)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Previously, the upload worker would skip uploading a file if there was a cached 'upload_successful' state, even if the cached state was from a completely different submission ID.
This change adds the 'submission_id' to the cache, ensuring that files are properly re-uploaded for new submissions.