Improve file integrity check + local testing environment#1045
Open
eachristgr wants to merge 3 commits into
Open
Improve file integrity check + local testing environment#1045eachristgr wants to merge 3 commits into
eachristgr wants to merge 3 commits into
Conversation
…wording The requests view read include_replicas/include_summary/include_logs as raw query strings, so "false" was truthy and the documented toggles silently did nothing — a regression from the UI PR (dmwm#1034) that dropped the string->bool coercion. Restore explicit coercion via a query_bool() helper; include_summary again defaults on for the detail view and off for the list view (where it costs one query per request). Also stop the submit response from advertising "Job ID: None." Jobs are now assigned asynchronously by the queue processor, so report the queued state honestly (status SUBMITTED, job_id null). Add regression tests for the boolean params (including HTML template rendering) and the submit wording.
…ygiene Make LOCAL_TESTING read from the environment (default False) and fall back to dev-default secrets + SQLite under it, so `LOCAL_TESTING=true manage.py test` runs with no source edits or .env file. Production behaviour is unchanged — the secrets are still required when LOCAL_TESTING is unset. Update the file_integrity_checker README: replace the manual settings-patch instructions with the LOCAL_TESTING workflow, fix the stale submit-response example and controller filenames, document UI access and deployment, and correct the test count. Ignore .env and *.sqlite3 (local secrets and the dev DB the workflow creates).
The per-RSE PFN loop appended a row on every PFN outcome, so an RSE with multiple PFNs could emit several rows. When the first PFN failed to copy but a later one succeeded, downstream get_or_create(request, lfn, rse) kept the first (ERROR) row and dropped the later OK — a transient copy failure became a sticky false verdict, and per-RSE counts were inflated. Extract the per-RSE logic into check_rse(), which tries PFNs in order, logs each attempt, validates the first that copies, and returns exactly one result per RSE (a single ERROR only if every PFN fails to copy). PFNs point at the same physical bytes, so re-checking other paths adds no value. The copy/checksum/ content calls are injectable to allow testing without the grid stack. Add test_run_check.py (7 unit tests, stdlib unittest, gfal2/uproot/rucio stubbed in sys.modules) covering the regression and the per-RSE behaviour, and document how to run them in the README.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes both DMOps apps testable without cluster secrets or a VOMS proxy, and tightens how the integrity checker handles a file's PFNs.
Local testing environment
Added a LOCAL_TESTING switch in settings.py. Set LOCAL_TESTING=true and the app runs self-contained — SQLite plus dummy secrets — so the suite runs with no .env and no source edits:
LOCAL_TESTING=true python manage.py testIt defaults to False, so production is untouched. On master, settings.py fails to import unless SECRET_KEY, JIRA_PAT, and GROUP_AUTHORIZATION_CLIENT_SECRET are set — which blocked every test, including the invalidation tool's fi_manager tests, from starting. LOCAL_TESTING=true supplies dummy values so the whole suite runs.
Integrity check