Automatic cleaning of input data Rucio rules

[nhduongvn]

Fixes #11409

Status

not-tested

Description

To automatically deleting the Rucio rules of input data set (block) of workflows, we implement a cherrypy thread that periodically scans the global workqueue for completed workqueue elements (percentSuccess == 100 and percentComplete == 100) and deletes the corresponding Rucio rules of input data block

There are also similar implementations in MSRuleCleaner (a new Pline for cleaning). However, Alan pointed out that MSRuleCleaner only runs on workflows at "later" stages ("announced"?). Therefore, the cleaning will not be done as soon as the processing of input data is complete. The code is still kept here (maybe for proof-of-concept for a new MS?).

Is it backward compatible (if not, which system it affects?)

YES

External dependencies / deployment changes

Yes, it requires the settings/configurations for a cherryPy thread that point to the Rucio account of input data set rules and the global queue

[dmwm-bot]

Jenkins results:

• Python3 Unit tests: failed
  • 4 new failures
  • 3 tests added
  • 4 changes in unstable tests
• Python3 Pylint check: failed
  • 17 warnings and errors that must be fixed
  • 9 warnings
  • 223 comments to review
• Pycodestyle check: succeeded
  • 230 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/940/artifact/artifacts/PullRequestReport.html

[dmwm-bot]

Jenkins results:

• Python3 Unit tests: failed
  • 4 new failures
  • 3 tests added
  • 3 changes in unstable tests
• Python3 Pylint check: failed
  • 1 warnings and errors that must be fixed
  • 9 warnings
  • 218 comments to review
• Pycodestyle check: succeeded
  • 225 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/941/artifact/artifacts/PullRequestReport.html

[nhduongvn]

These are failed unit tests which not related to the new codes:
WMCore_t.ReqMgr_t.Service_t.Auxiliary_t.AuxiliaryTest:testAbout changed from success to error
WMCore_t.ReqMgr_t.Service_t.Auxiliary_t.AuxiliaryTest:testType changed from success to error
WMCore_t.ReqMgr_t.Service_t.Auxiliary_t.AuxiliaryTest:testUnifiedConfig changed from success to error
WMCore_t.ReqMgr_t.Service_t.Auxiliary_t.AuxiliaryTest:testWMAgentConfig changed from success to error
This PR branch already has all latest updates (I ran git fetch upstream).

[nhduongvn]

please test

[dmwm-bot]

Jenkins results:

• Python3 Unit tests: failed
  • 6 new failures
  • 2 tests added
  • 2 changes in unstable tests
• Python3 Pylint check: failed
  • 1 warnings and errors that must be fixed
  • 1 warnings
  • 109 comments to review
• Pycodestyle check: succeeded
  • 141 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/1097/artifact/artifacts/PullRequestReport.html

[dmwm-bot]

Jenkins results:

• Python3 Unit tests: failed
  • 4 new failures
  • 2 tests added
  • 5 changes in unstable tests
• Python3 Pylint check: failed
  • 8 warnings and errors that must be fixed
  • 1 warnings
  • 143 comments to review
• Pycodestyle check: succeeded
  • 189 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/1126/artifact/artifacts/PullRequestReport.html

[dmwm-bot]

Jenkins results:

• Python3 Unit tests: failed
  • 5 new failures
  • 2 tests added
  • 2 changes in unstable tests
• Python3 Pylint check: failed
  • 33 warnings and errors that must be fixed
  • 6 warnings
  • 185 comments to review
• Pycodestyle check: succeeded
  • 335 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/WMCore-PR-Report/1198/artifact/artifacts/PullRequestReport.html

[nhduongvn]

retest

[nhduongvn]

rebuild

[nhduongvn]

All 4 errors are pre-existing issues below unrelated to changes in this PR:
testReplaceToSantizeURL — bug in Lexicon.py (regex syntax)
testWMStatsWriter — API signature mismatch in WMStatsReader
testFailStepChainACDC — external CouchDB network timeout
testHeadGzip — external service timeout
See https://cmssdt.cern.ch/dmwm-jenkins/job/WMCore-PR-Test/1232/console

[hassan11196]

test this please