Skip to content

OCPNODE-4383: prom rules: add alert for nodes using runc#5874

Open
haircommander wants to merge 2 commits intoopenshift:mainfrom
haircommander:runc-alert
Open

OCPNODE-4383: prom rules: add alert for nodes using runc#5874
haircommander wants to merge 2 commits intoopenshift:mainfrom
haircommander:runc-alert

Conversation

@haircommander
Copy link
Copy Markdown
Member

@haircommander haircommander commented Apr 22, 2026

relies on a cri-o metric that's emitted when the cluster is using runc
generated with claude

Summary by CodeRabbit

  • New Features
    • Added an informational alert (RuncDeprecated) that notifies when the deprecated runc runtime is detected for 10 minutes; includes summary, migration guidance, and a runbook link.
    • Extended CRI-O metrics collection to include a new "default_runtime" metric collector on control-plane and worker nodes.

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Apr 22, 2026
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

openshift-ci-robot commented Apr 22, 2026

@haircommander: This pull request references OCPNODE-4446 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

relies on a cri-o metric that's emitted when the cluster is using runc
generated with claude

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Apr 22, 2026

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 46ab94c3-e9b6-4c37-9b4a-1d6987ce5456

📥 Commits

Reviewing files that changed from the base of the PR and between ee72fc1 and 6c48b96.

📒 Files selected for processing (4)
  • install/0000_90_machine-config_01_prometheus-rules.yaml
  • templates/arbiter/01-arbiter-container-runtime/_base/files/crio.yaml
  • templates/master/01-master-container-runtime/_base/files/crio.yaml
  • templates/worker/01-worker-container-runtime/_base/files/crio.yaml
✅ Files skipped from review due to trivial changes (1)
  • templates/arbiter/01-arbiter-container-runtime/_base/files/crio.yaml
🚧 Files skipped from review as they are similar to previous changes (3)
  • templates/worker/01-worker-container-runtime/_base/files/crio.yaml
  • templates/master/01-master-container-runtime/_base/files/crio.yaml
  • install/0000_90_machine-config_01_prometheus-rules.yaml

Walkthrough

Added a new Prometheus alert group runc-deprecated with alert RuncDeprecated that fires when count(container_runtime_crio_default_runtime{runtime="runc"}) > 0 for 10m, and added the default_runtime metrics collector to CRI-O configs for arbiter, master, and worker.

Changes

Cohort / File(s) Summary
Prometheus Alert Rule
install/0000_90_machine-config_01_prometheus-rules.yaml
Added new alert group runc-deprecated containing RuncDeprecated which triggers on count(container_runtime_crio_default_runtime{runtime="runc"}) > 0 for 10m; includes labels namespace: openshift-machine-config-operator, severity: info, and annotations summary, description, runbook_url.
CRI-O Metrics Configs
templates/arbiter/01-arbiter-container-runtime/_base/files/crio.yaml, templates/master/01-master-container-runtime/_base/files/crio.yaml, templates/worker/01-worker-container-runtime/_base/files/crio.yaml
Added the "default_runtime" entry to [crio.metrics].metrics_collectors alongside the existing "processes_defunct" in arbiter, master, and worker CRI-O configurations.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 12
✅ Passed checks (12 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title directly and accurately summarizes the main change: adding a Prometheus alert rule to detect nodes using runc.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The PR modifies only YAML configuration files (Prometheus rules and CRI-O settings), not Ginkgo test code. No test files are added or modified.
Test Structure And Quality ✅ Passed This check is not applicable to the pull request. The PR modifies only declarative configuration files (Prometheus alert rule in YAML and CRI-O metrics configuration), not test files.
Microshift Test Compatibility ✅ Passed This PR does not introduce any Ginkgo e2e tests. The changes consist solely of configuration files: a Prometheus alert rule YAML file and three CRI-O configuration template files.
Single Node Openshift (Sno) Test Compatibility ✅ Passed This pull request does not add any Ginkgo e2e tests. The PR modifies only YAML configuration files: a Prometheus alert rules file and three CRI-O configuration files. Since the custom check specifically applies to "When new Ginkgo e2e tests are added," and no such tests exist in this PR, the check is not applicable and passes by default.
Topology-Aware Scheduling Compatibility ✅ Passed The PR introduces Prometheus alert rules and CRI-O metric collectors; no pod deployments, scheduling constraints, affinity rules, or topology assumptions are added.
Ote Binary Stdout Contract ✅ Passed PR modifies only configuration files (Prometheus rules and CRI-O YAML configs); no executable Go code entry points are introduced or modified.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed No new Ginkgo e2e tests are being added. The PR only modifies YAML configuration and CRI-O template files.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci openshift-ci Bot requested a review from djoshy April 22, 2026 18:26
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Apr 22, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: haircommander
Once this PR has been reviewed and has the lgtm label, please assign yuqi-zhang for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@install/0000_90_machine-config_01_prometheus-rules.yaml`:
- Around line 29-30: Fix the grammar typo in the Prometheus alert rule
description: update the description field value (the YAML key "description" in
the rule) to change “support will removed” to “support will be removed” so the
user-facing message reads correctly.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 199b41d8-1f05-4044-a1a5-3bacba2e627a

📥 Commits

Reviewing files that changed from the base of the PR and between c3a9db7 and 7e20fb5.

📒 Files selected for processing (1)
  • install/0000_90_machine-config_01_prometheus-rules.yaml

Comment thread install/0000_90_machine-config_01_prometheus-rules.yaml Outdated
@haircommander haircommander changed the title OCPNODE-4446: prom rules: add alert for nodes using runc OCPNODE-4382: prom rules: add alert for nodes using runc Apr 22, 2026
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

openshift-ci-robot commented Apr 22, 2026

@haircommander: This pull request references OCPNODE-4382 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target the "5.0.0" version, but no target version was set.

Details

In response to this:

relies on a cri-o metric that's emitted when the cluster is using runc
generated with claude

Summary by CodeRabbit

  • New Features
  • Added an informational alert that detects use of the deprecated runc container runtime and fires after sustained detection (10 minutes).
  • Alert includes a clear summary, detailed migration guidance, and a runbook link to assist operators.
  • Targets the machine-config-operator area and is additive (no existing alerts changed).

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@haircommander haircommander changed the title OCPNODE-4382: prom rules: add alert for nodes using runc OCPNODE-4383: prom rules: add alert for nodes using runc Apr 22, 2026
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

openshift-ci-robot commented Apr 22, 2026

@haircommander: This pull request references OCPNODE-4383 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

relies on a cri-o metric that's emitted when the cluster is using runc
generated with claude

Summary by CodeRabbit

  • New Features
  • Added an informational alert that detects use of the deprecated runc container runtime and fires after sustained detection (10 minutes).
  • Alert includes a clear summary, detailed migration guidance, and a runbook link to assist operators.
  • Targets the machine-config-operator area and is additive (no existing alerts changed).

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Apr 22, 2026

Caution

Failed to replace (edit) comment. This is likely due to insufficient permissions or the comment being deleted.

Error details
{}

@haircommander
Copy link
Copy Markdown
Member Author

/retest

1 similar comment
@haircommander
Copy link
Copy Markdown
Member Author

/retest

relies on a cri-o metric that's emitted when the cluster is using runc

Signed-off-by: Peter Hunt <pehunt@redhat.com>
Signed-off-by: Peter Hunt <pehunt@redhat.com>
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Apr 24, 2026

@haircommander: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Copy link
Copy Markdown
Contributor

@yuqi-zhang yuqi-zhang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall seems fine, just some questions in line

The runc OCI runtime has been deprecated and support will be removed in a future release. Migrate to crun before upgrading to a future OpenShift release.
See https://docs.redhat.com/en/documentation/openshift_container_platform/latest/html/machine_configuration/machine-configs-custom#config-container-runtime_machine-configs-custom
for migration steps.
runbook_url: https://github.com/openshift/runbooks/blob/master/alerts/machine-config-operator/RuncDeprecated.md
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this still in progress? I don't see the file or a PR yet.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

annotations:
summary: "This cluster is using the deprecated runc container runtime"
description: >-
The runc OCI runtime has been deprecated and support will be removed in a future release. Migrate to crun before upgrading to a future OpenShift release.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds like this should be an upgradeable=false condition instead?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we won't be blocking the upgrade, especially 4.21->4.22. in the 5.0 cycle we will block RCHOS upgrade from 9 to 10 if runc is in use. this is just the warning

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, this is merging into 5.0 branch right now. Are you looking to backport this?

I'm fine with doing the blocking edge as a followup but https://redhat.atlassian.net/browse/OCPNODE-4446 targets 4.22, so maybe it'd be better as a bug?

(also I'm not sure if we have a conditional block for 9->10 specifically yet, since the expectation is that you'd be switching at the pool level whenever you want, and not as part of an OCP upgrade, so we'd have to come up with a new mechanism there)

rules:
- alert: RuncDeprecated
expr: |
count(container_runtime_crio_default_runtime{runtime="runc"}) > 0
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess there's no scenario to have different default runtimes on different nodes (can you even do that?) so this is probably a fine way to evaluate it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there could be (different MCPs can have different ones) but it's reported once on each node, and all we care about is no node has runc as default

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, ok, I thought you had to have it homogeneous across the cluster, I guess now?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants