STF 1.5.7 release ops (#697)

* Update ansible-core in CI (#662)

Updates the ansible-core version to 2.17.6 to fix several vulnerabilities existing in the current version (ansible-core==2.15.9).

By default, Python3.9 is available in Centos 9 Stream. Unfortunately, newer versions of ansible-core are not available for Python3.9, so this change also updates our CI to install and consume Python 3.12.

Also use kubernetes.core 5.0.0 in CI, we are already building STO with this version. Related commit a2ec384

* Fix CI readme

Jobs now run on OCP 4.16 and OCP 4.18. And we were missing the nighly bundles + index deploy jobs index description

* Fix Ansible-Core vulnerable to content protections bypass (#664)

A flaw was found in Ansible-Core. This vulnerability allows attackers
to bypass unsafe content protections using the hostvars object to reference
and execute templated content. This issue can lead to arbitrary code
execution if remote data or module outputs are improperly templated within
playbooks.

Affected versions >= 2.17.0b1, < 2.17.7rc1
Patched version 2.17.7rc1

* Use operator-sdk v1.39.2 (#666)

Using a newer operator-sdk will make dependency management a lot easier
for downstream hermetic build systems, like the ones used downstream.

It also lets us do away with using different versions for bundle
generation and deployment in our CI.

* Close stdin on workflow operator-sdk generate bundle calls

An open stdin makes operator-sdk assume it's supposed to take input from
stdin, when we actually want it to look at the working directory. There
is no way to avoid this using command-line flags, so we need to
explicitly close the stdin pipe in all calls to it.

* Bundle changes for Konflux support (#669)

* Use full pullspecs for related images

Renovate, used by Konflux for updating dependency references, works
better with full container pullspecs rather than separate repository
URLs and tags.

This lets us inject pullspecs pinned by digest, which don't quite follow
the repo:tag syntax.

* Don't set update graph metadata in CSV

This was originally added for the benefit of downstream builds, but
those are moving to file-based catalog fragments which don't need update
graph metadata to be embedded in the bundle.

* Refactor pullspec building to make it more readable

* Use /tmp instead of the base_dir to fetch CSV files

* Update selector to get builder-dockercfg secret name (#674)

Fix the step to properly gather the builder-dockercfg secret name. This is required to correctly set up the auth when building the index image

* Use sto/sgo_bundle_image_path_internal for the index (#675)

sto_bundle_image_path and sg_bundle_image_path variable names
are used already by some parts of the automation and there is
an overlap

* Refactor steps for getting bundles info for local builds

* Drop when clause for "Generate default package names" step

We already check the condition holds for the entire block

* Bump requests from 2.32.0 to 2.32.4 in /build/stf-run-ci (#672)

Bumps [requests](https://github.com/psf/requests) from 2.32.0 to 2.32.4.
- [Release notes](https://github.com/psf/requests/releases)
- [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md)
- [Commits](psf/requests@v2.32.0...v2.32.4)

---
updated-dependencies:
- dependency-name: requests
  dependency-version: 2.32.4
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Victoria Martinez de la Cruz <victoria@redhat.com>

* Set permissions explicitly to actions job (#676)

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* bundle gen: allow passing in pre-generated bundle version (#678)

This improves our support for generating bundles as part of multistage
container builds, as a single pre-generated bundle version can be fed
into the generated bundle and into the bundle container labels.

RELDEL-7554

* Use crc-cloud image for OCP 4.18 jobs (#680)

coreos-crc-extracted-2-48-0-3xl is deprecated, the
replacement for it is crc-cloud-ocp-4-18-1-3xl

Available images are in https://review.rdoproject.org/zuul/labels

* Add support for deploy from catalog

Add __deploy_from_catalog and __stf_catalog_index_image_path options
to stf-run-ci.

This new option allows to create a CatalogSource using a pre-built
index image and deploy STF from it.

This is particularly useful to test package metadata issues that would
be invisible when testing bundles but very apparent when installing
from the actual catalog.

Closes: OSPRH-19028

* Update ansible-lint-ignore and fix linting issues

We had some pre-existing linting issues that reappeared when renaming
create_catalog.yml to create_index_image.yml. Update ansible-lint-ignore
to supress these warnings.

Also fixes create_catalog.yml linting issue (missing empty line at EOF)

* Update olm.maxOpenShiftVersion to 4.18

olm.maxOpenShiftVersion in STO should be 4.18 instead of 4.16

This wrongly set config is preventing STF users to easily perform
upgrades to the latest OCP supported version

Closes-Bug: OSPRH-18670

* [zuul] Add PR jobs to the periodic lines

Since there is less development in STF now, we don't run the PR jobs as
often. Running the jobs reqularly helps to find issues that crop up due
to changing deps, etc and make sure the jobs still work.

This change adds the jobs from the github-check pipeline into the
periodic line, ensuring that they run regularly.

* Use include_tasks instead of include_role

* Drop name from include_tasks

* Use catalogSource name and namespace for subscriptions

Update setup_stf to use infrawatch_operators and namespace
variables instead of hardwiring them.

We have some logic that could go away now that we can select
to deploy from a specific catalogsource. Precisely,
infrawatch_catalog_source_manifest param. This was being used
to pass a catalogsource manifest and deploy from it.

* Drop "Set default InfraWatch OperatorSource manifest"

This step is not longer useful since we can deploy from catalog
by passing a pre-built index image or by using the default
(which corresponds to the default in use here)

* Clean up setup and deploy logic

* Fix typo in catalog name for infrawatch_operators

* Add jobs for deploy_from_catalog (#685)

Add two jobs to test the deploy_from_catalog option
By default, it will use the index image in
https://quay.io/repository/infrawatch-operators/infrawatch-catalog

Test deploying from the two latest supported OCP versions

Closes: https://issues.redhat.com/browse/OSPRH-19031

* Enable all check jobs

* Update ansible-lint pinned version (#688)

Bump ansible-lint to 25.2.0

* Update ansible-lint-ignore with new linting issues

ansible-lint 25.2.0 introduces new linting rules that
we are not following. Add them to the ignore list.

* Pin to ansible 12.0.0

* Fix stf-run-ci mutually exclusive deployment flags (#686)

Fail when two or more of the non compatible deployment
options are set. To make the check simpler, convert them
to integers (where true becomes 1 and false becomes 0)
and check if their sum is greater than 1.

Closes-Bug: OSPRH-20165

* Use openshift-ansible-operator 4.20 in STO (#695)

Partially-Closes: OSPRH-21875

* Bump the major supported OCP version for STO to OCP 4.20 (#693)

* Bump the major supported OCP version for STO to OCP 4.20

* Drop OCP 4.16 from Dockerfile

* Add OCP 4.20 jobs (#694)

Add job definitions using crc-cloud-ocp-4-20-1-3xl as base image

* Update nodesets name to stf-crc_cloud-ocpX.Y

We no longer use crc extracted images, we now use crc cloud images

Update the naming accordingly

* Drop OCP 4.16 jobs

Closes: OSPRH-21881
Closes: OSPRH-21882

* Fix typo in build/stf-run-ci/README.md (#698)

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: migarcia <migarcia@redhat.com>
Co-authored-by: Emma Foley <elfiesmelfie@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Emma Foley <efoley@redhat.com>
Co-authored-by: Alex Yefimov <126113326+ayefimov-1@users.noreply.github.com>

Author: 6 people

.ansible-lint-ignore

@@ -50,20 +50,22 @@ roles/servicetelemetry/tasks/pre.yml warning[outdated-tag]
 roles/servicetelemetry/vars/dummy_user_certs.yml var-naming[no-role-prefix]
 
 # build/stf-run-ci
+# This file contains ignores rule violations for ansible-lint
 build/stf-run-ci/defaults/main.yml var-naming[no-reserved]
 build/stf-run-ci/handlers/main.yml yaml[new-line-at-end-of-file]
 build/stf-run-ci/tasks/clone_repos.yml name[template]
 build/stf-run-ci/tasks/clone_repos.yml yaml[empty-lines]
 build/stf-run-ci/tasks/create_builds.yml jinja[invalid]
 build/stf-run-ci/tasks/create_builds.yml name[missing]
 build/stf-run-ci/tasks/create_builds.yml no-changed-when
-build/stf-run-ci/tasks/create_catalog.yml ignore-errors
 build/stf-run-ci/tasks/create_catalog.yml jinja[invalid]
-build/stf-run-ci/tasks/create_catalog.yml jinja[spacing]
-build/stf-run-ci/tasks/create_catalog.yml name[missing]
-build/stf-run-ci/tasks/create_catalog.yml no-changed-when
-build/stf-run-ci/tasks/create_catalog.yml risky-file-permissions
-build/stf-run-ci/tasks/create_catalog.yml yaml[truthy]
+build/stf-run-ci/tasks/create_index_image.yml ignore-errors
+build/stf-run-ci/tasks/create_index_image.yml jinja[invalid]
+build/stf-run-ci/tasks/create_index_image.yml jinja[spacing]
+build/stf-run-ci/tasks/create_index_image.yml name[missing]
+build/stf-run-ci/tasks/create_index_image.yml no-changed-when
+build/stf-run-ci/tasks/create_index_image.yml risky-file-permissions
+build/stf-run-ci/tasks/create_index_image.yml yaml[truthy]
 build/stf-run-ci/tasks/main.yml jinja[invalid]
 build/stf-run-ci/tasks/main.yml jinja[spacing]
 build/stf-run-ci/tasks/main.yml key-order[task]
@@ -82,7 +84,7 @@ build/stf-run-ci/tasks/setup_elasticsearch.yml jinja[invalid]
 build/stf-run-ci/tasks/setup_elasticsearch.yml yaml[new-line-at-end-of-file]
 build/stf-run-ci/tasks/setup_registry_auth.yml jinja[invalid]
 build/stf-run-ci/tasks/setup_registry_auth.yml no-changed-when
-build/stf-run-ci/tasks/setup_stf.yml yaml[trailing-spaces]
+build/stf-run-ci/tasks/setup_stf.yml jinja[invalid]
 build/stf-run-ci/tasks/setup_stf_from_bundles.yml command-instead-of-shell
 build/stf-run-ci/tasks/setup_stf_from_bundles.yml no-changed-when
 build/stf-run-ci/tasks/setup_stf_local_build.yml command-instead-of-module

.github/workflows/main.yml

@@ -1,5 +1,7 @@
 name: CI
 on: push
+permissions:
+  contents: read
 
 jobs:
   operator-linting:

.zuul.yaml

@@ -1,19 +1,19 @@
 ---
 - nodeset:
-    name: stf-crc_extracted-ocp416
+    name: stf-crc_cloud-ocp418
     nodes:
       - name: controller
         label: cloud-centos-9-stream-tripleo-vexxhost
       - name: crc
-        label: coreos-crc-extracted-2-39-0-3xl
+        label: crc-cloud-ocp-4-18-1-3xl
 
 - nodeset:
-    name: stf-crc_extracted-ocp418
+    name: stf-crc_cloud-ocp420
     nodes:
       - name: controller
         label: cloud-centos-9-stream-tripleo-vexxhost
       - name: crc
-        label: crc-cloud-ocp-4-18-1-3xl
+        label: crc-cloud-ocp-4-20-1-3xl
 
 - job:
     name: stf-base-2node
@@ -134,86 +134,112 @@
       scenario: "nightly_bundles-index_deploy"
 
 - job:
-    name: stf-crc-ocp_416-nightly_bundles
-    parent: stf-crc-nightly_bundles
+    name: stf-crc-catalog_deploy
+    parent: stf-base
+    abstract: true
     description: |
-      Deploy STF using the nightly bundles on OCP 4.16
-    nodeset: stf-crc_extracted-ocp416
+      Use a pre-built index image to create a CatalogSource and deploy STF from it
+    vars:
+      scenario: "catalog_deploy"
 
 - job:
     name: stf-crc-ocp_418-nightly_bundles
     parent: stf-crc-nightly_bundles
     description: |
       Deploy STF using the nightly bundles on OCP 4.18
-    nodeset: stf-crc_extracted-ocp418
+    nodeset: stf-crc_cloud-ocp418
 
 - job:
-    name: stf-crc-ocp_416-local_build
-    parent: stf-crc-local_build
+    name: stf-crc-ocp_420-nightly_bundles
+    parent: stf-crc-nightly_bundles
     description: |
-      Build STF locally and deploy from index on OCP 4.16
-    nodeset: stf-crc_extracted-ocp416
+      Deploy STF using the nightly bundles on OCP 4.20
+    nodeset: stf-crc_cloud-ocp420
 
 - job:
     name: stf-crc-ocp_418-local_build
     parent: stf-crc-local_build
     description: |
       Build STF locally and deploy from index on OCP 4.18
-    nodeset: stf-crc_extracted-ocp418
+    nodeset: stf-crc_cloud-ocp418
 
 - job:
-    name: stf-crc-ocp_416-local_build-index_deploy
-    parent: stf-crc-local_build-index_deploy
+    name: stf-crc-ocp_420-local_build
+    parent: stf-crc-local_build
     description: |
-      Build STF locally and deploy from index on OCP 4.16
-    nodeset: stf-crc_extracted-ocp416
+      Build STF locally and deploy from index on OCP 4.20
+    nodeset: stf-crc_cloud-ocp420
 
 - job:
     name: stf-crc-ocp_418-local_build-index_deploy
     parent: stf-crc-local_build-index_deploy
     description: |
       Build STF locally and deploy from index on OCP 4.18
-    nodeset: stf-crc_extracted-ocp418
+    nodeset: stf-crc_cloud-ocp418
 
 - job:
-    name: stf-crc-ocp_416-nightly_bundles-index_deploy
-    parent: stf-crc-nightly_bundles-index_deploy
+    name: stf-crc-ocp_420-local_build-index_deploy
+    parent: stf-crc-local_build-index_deploy
     description: |
-      Create an index image using nightly bundles and deploy STF from it on OCP 4.16
-    nodeset: stf-crc_extracted-ocp416
+      Build STF locally and deploy from index on OCP 4.20
+    nodeset: stf-crc_cloud-ocp420
 
 - job:
     name: stf-crc-ocp_418-nightly_bundles-index_deploy
     parent: stf-crc-nightly_bundles-index_deploy
     description: |
       Create an index image using nightly bundles and deploy STF from it on OCP 4.18
-    nodeset: stf-crc_extracted-ocp418
+    nodeset: stf-crc_cloud-ocp418
+
+- job:
+    name: stf-crc-ocp_420-nightly_bundles-index_deploy
+    parent: stf-crc-nightly_bundles-index_deploy
+    description: |
+      Create an index image using nightly bundles and deploy STF from it on OCP 4.20
+    nodeset: stf-crc_cloud-ocp420
+
+- job:
+    name: stf-crc-ocp_418-catalog_deploy
+    parent: stf-crc-catalog_deploy
+    description: |
+      Use a pre-built index image to create a CatalogSource and deploy STF from it on OCP 4.18
+    nodeset: stf-crc_cloud-ocp418
+
+- job:
+    name: stf-crc-ocp_420-catalog_deploy
+    parent: stf-crc-catalog_deploy
+    description: |
+      Use a pre-built index image to create a CatalogSource and deploy STF from it on OCP 4.20
+    nodeset: stf-crc_cloud-ocp420
+
 
 - project-template:
     name: stf-crc-jobs
     description: |
       STF CRC jobs that build and deploy STF
     github-check:
       jobs:
-        - stf-crc-ocp_416-local_build
         - stf-crc-ocp_418-local_build
-        - stf-crc-ocp_416-local_build-index_deploy
+        - stf-crc-ocp_420-local_build
         - stf-crc-ocp_418-local_build-index_deploy
-        - stf-crc-ocp_416-nightly_bundles-index_deploy
+        - stf-crc-ocp_420-local_build-index_deploy
         - stf-crc-ocp_418-nightly_bundles-index_deploy
+        - stf-crc-ocp_420-nightly_bundles-index_deploy
+        - stf-crc-ocp_418-catalog_deploy
+        - stf-crc-ocp_420-catalog_deploy
 
 - project:
     name: infrawatch/service-telemetry-operator
     templates:
       - stf-crc-jobs
     periodic:
       jobs:
-        - stf-crc-ocp_416-nightly_bundles
         - stf-crc-ocp_418-nightly_bundles
-        - stf-crc-ocp_416-local_build
+        - stf-crc-ocp_420-nightly_bundles
         - stf-crc-ocp_418-local_build
-        - stf-crc-ocp_416-local_build-index_deploy
+        - stf-crc-ocp_420-local_build
         - stf-crc-ocp_418-local_build-index_deploy
-        - stf-crc-ocp_416-nightly_bundles-index_deploy
+        - stf-crc-ocp_420-local_build-index_deploy
+        - stf-crc-ocp_418-nightly_bundles-index_deploy
         - stf-crc-ocp_418-nightly_bundles-index_deploy
 

build/Dockerfile

@@ -1,4 +1,4 @@
-FROM quay.io/operator-framework/ansible-operator:v1.36.1
+FROM quay.io/operator-framework/ansible-operator:v1.38.1
 
 # temporarily switch to root user to adjust image layers
 USER 0

build/stf-run-ci/README.md

@@ -21,9 +21,11 @@ choose to override:
 | `__local_build_enabled`                                              | {true,false}                                                | true                                                                          | Whether to deploy STF from local built artifacts. Also see `working_branch`, `sg_branch`, `sgo_branch`                                                                                                                  |
 | `__deploy_from_bundles_enabled`                                      | {true,false}                                                | false                                                                         | Whether to deploy STF from OLM bundles (TODO: compat with `__local_build_enabled`)                                                                                                                                      |
 | `__deploy_from_index_enabled`                                        | {true,false}                                                | false                                                                         | Whether to deploy STF from locally built bundles/OLM bundles and index image.                                                                                                                                                       |
+| `__deploy_from_catalog`                                              | {true,false}                                                | false                                                                         | Whether to deploy STF from a pre-built index image.                                                                                                                                                                    |
 | `__disconnected_deploy`                                              | {true,false}                                                | false                                                                         | Whether to deploy on a disconnected cluster                                                                                                                                                                               |
 | `__service_telemetry_bundle_image_path`                              | <image_path>                                                | `quay.io/infrawatch-operators/service-telemetry-operator-bundle:nightly-head` | Image path to Service Telemetry Operator bundle                                                                                                                                                                         |
 | `__smart_gateway_bundle_image_path`                                  | <image_path>                                                | `quay.io/infrawatch-operators/smart-gateway-operator-bundle:nightly-head`     | Image path to Smart Gateway Operator bundle                                                                                                                                                                             |
+| `__stf_catalog_index_image_path`                                     | <image_path>                                                | `quay.io/infrawatch-operators/infrawatch-catalog:nightly`                     | Index image path to STF catalog                                                                                                                                                                                              |
 | `setup_bundle_registry_tls_ca`                                       | {true,false}                                                | true                                                                          | Whether to setup or not a TLS CA cert for the bundle registry access                                                                                                                                                    |
 | `setup_bundle_registry_auth`                                         | {true,false}                                                | true                                                                          | Whether to setup or not the auth for the bundle registry access                                                                                                                                                         |
 | `prometheus_webhook_snmp_branch`                                     | <git_branch>                                                | master                                                                        | Which Prometheus Webhook SNMP git branch to checkout                                                                                                                                                                    |
@@ -135,7 +137,7 @@ You can perform a deployment using OLM and a Subscription from locally built art
 ansible-playbook -e __local_build_enabled=true -e __deploy_from_index_enabled=true run-ci.yaml
 ```
 
-## Deployment with pre-build bundles and index
+## Deployment with pre-built bundles and locally created index image
 
 Instead of relying on the operator-sdk to deploy from selected bundles using the "operator-sdk run bundle" utility,
 you can perform a deployment using OLM and a Subscription to a locally created index image like this:
@@ -155,6 +157,19 @@ Since you will fetch the selected images from a bundle registry, it is required
 access credentials for the desired registry correctly configured. Check the "Deployment with pre-build bundles"
 docs above to get more information about this.
 
+## Deployment with pre-built index image
+
+It is also possible to pass the path of a pre-built index image to create a CatalogSource and deploy STF from it.
+This can be done as follows:
+
+```sh
+ansible-playbook -e __local_build_enabled=false -e __deploy_from_catalog=true \
+  -e __stf_catalog_index_image_path=<registry>/<namespace>/infrawatch-catalog:<tag> \
+  -e pull_secret_registry=<registry> \
+  -e pull_secret_user=<username> \
+  -e pull_secret_pass=<password>
+  run-ci.yaml
+```
 
 # License
 

build/stf-run-ci/defaults/main.yml

@@ -8,6 +8,7 @@ list_of_stf_objects:
 __local_build_enabled: true
 __deploy_from_bundles_enabled: false
 __deploy_from_index_enabled: false
+__deploy_from_catalog: false
 __deploy_stf: true
 
 __disconnected_deploy: false
@@ -34,6 +35,7 @@ __service_telemetry_transports_certificates_ca_cert_duration: 70080h
 __internal_registry_path: image-registry.openshift-image-registry.svc:5000
 __service_telemetry_bundle_image_path: "quay.io/infrawatch-operators/service-telemetry-operator-bundle:nightly-head"
 __smart_gateway_bundle_image_path: "quay.io/infrawatch-operators/smart-gateway-operator-bundle:nightly-head"
+__stf_catalog_index_image_path: "quay.io/infrawatch-operators/infrawatch-catalog:nightly"
 
 default_operator_registry_image_base: registry.redhat.io/openshift4/ose-operator-registry
 default_operator_registry_image_tag: v4.13

build/stf-run-ci/requirements.txt

@@ -1,5 +1,5 @@
 # https://stackoverflow.com/questions/64073422/importerror-cannot-import-name-oauth1session-from-requests-oauthlib
-requests==2.32.0
+requests==2.32.4
 requests_oauthlib==1.3.0
 # https://github.com/domainaware/parsedmarc/issues/318
 oauthlib==3.2.2