Monitoring op scripts

[juanpablosalas]

Migration monitoring cronjobs deployment (Fixes #904)

This PR addresses the implementation discussed in #904 by adding the monitoring cronjobs that are currently executed manually on the integration environment.

What changed

Monitoring scripts

Added the docker/monitoring directory, containing the source code for the monitoring cronjobs. The directory is an exact copy of the relevant scripts from the existing monitoring repository ([3], [4], [5]).

The following cronjobs are included:

rucio-account-usage
rucio-account-usage-locks
rucio-activity-account-usage
rucio-activity-usage
rucio-errors-volume
rucio-locks-count-volume
rucio-rules-count-volume
rucio-subscriptions
rse-usage
rucio-summary

Helm chart

Updated the helm/rucio-cron-jobs chart to deploy the new monitoring jobs, including:

• CronJob templates
• Services
• PersistentVolumeClaim (PVC)
• StorageClass

The chart version has been bumped to 4.1.0.

This work continues the deployment effort started in #910.

CI/CD

Added a GitHub Actions workflow to automatically build and publish the cmsrucio/dmops-spark-crons image.

The workflow is triggered by releases whose tag starts with: monitoring-

References

[1] CMSTRANSF-1366
[2] CMSTRANSF-1264
[3] https://gitlab.cern.ch/cmsdmops/cmsdmops/-/merge_requests/28/diffs
[4] https://gitlab.cern.ch/cmsdmops/cmsdmops/-/merge_requests/25/diffs
[5] https://gitlab.cern.ch/cmsdmops/cmsdmops/-/tree/master/monitoring/kubernetes?ref_type=heads

[ericvaandering]

I don't have any comments on specific scripts, but on overall structure and some python stuff (which applies to multiple files, not just where it is noted)

[ericvaandering]

What's in this file?

[juanpablosalas]

See comment below.

[ericvaandering]

What's in here. I want to make sure these binary files don't contain secrets of some sort.

[juanpablosalas]

These files (cmsmonitoring.zip and stomp.zip) contain helper scripts to submit the spark jobs. I tried to replace them with the zip files provisioned by the base image [1], but unfortunately this made the jobs fail. I assume these files were added in order to patch these failures so they remain necessary.

[1] https://github.com/dmwm/CMSKubernetes/blob/b77906f3933ace456e3af9dd7b5101eab8388e2f/docker/cmsmon-spark/install.sh#L26

[ericvaandering]

I think it makes sense to name this file something more specific that services. Should another group of pods need services, I'd like not to just keep expanding this.

Also is a service really needed for all these? They all have to be contacted by processes inside the cluster?

[juanpablosalas]

Renamed to dmtops-cj-spark-services.yaml.

Yes, these services are necessary since the cronjobs submit Spark jobs to an external cluster, and the ports are needed for the jobs to communicate back. A separate service/port is used for each one of the cronjobs in order to prevent clashes between the jobs.