Add policy for node image lifecycle management - #558
Open
DustinTrap wants to merge 6 commits into
Open
Conversation
OpenShift's default kubelet image garbage collection is driven by disk usage thresholds (85%/80%) and does not proactively remove stale images after cluster upgrades. On nodes with large disks, old image versions from prior OCP releases can accumulate indefinitely, causing image-based security scanners to report false positives on vulnerabilities that have already been patched in the running containers. This policy provides two mechanisms: 1. KubeletConfig: Lowers GC thresholds to 65%/50% on worker nodes so disk-pressure-based cleanup triggers earlier and cleans more aggressively. 2. CronJob: Runs weekly on worker nodes and removes images that are both unreferenced by any container and older than 7 days. The age filter protects images from workloads temporarily scaled to zero. Validated on OCP 4.22.7 with ACM 2.17.0. Compatible with OCP 4.8+ and ACM 2.4+. Signed-off-by: Dustin Trapani <12038788+DustinTrap@users.noreply.github.com> Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Dustin Trapani <12038788+DustinTrap@users.noreply.github.com>
|
@DustinTrap: GitHub didn't allow me to request PR reviews from the following users: grimsleydl. Note that only open-cluster-management-io members and repo collaborators can review this PR, and authors cannot review their own PRs. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: DustinTrap The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds an OCM policy that provisions hub-configured kubelet image garbage-collection settings and installs scheduled worker-node pruning for old, unreferenced CRI images. The policy is also added to the Configuration Management catalog with an OpenShift 4.8+ prerequisite. ChangesNode image lifecycle management
Estimated code review effort: 3 (Moderate) | ~25 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (2)
community/CM-Configuration-Management/policy-node-image-lifecycle.yaml (2)
153-153: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAvoid the
:latesttag; pin to a specific version.This repo already publishes a Gatekeeper policy (
policy-gatekeeper-container-image-latest.yaml) specifically to flag containers using the_latest_tag as non-compliant. Usingregistry.redhat.io/openshift4/ose-cli:latesthere is inconsistent with that stance and makes the pruning job's behavior non-reproducible across runs as the image is updated upstream.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@community/CM-Configuration-Management/policy-node-image-lifecycle.yaml` at line 153, Replace the :latest tag in the policy-node-image-lifecycle image reference with a specific, immutable ose-cli version tag, preserving the existing registry and repository while making pruning job runs reproducible.
136-136: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winConsider
concurrencyPolicy: ForbidgivenbackoffLimit: 0.With
Allow, a slow-running prune job could overlap with the next scheduled run, causing twocrictl rmiinvocations to race on the same node. Since retries are disabled (backoffLimit: 0), overlapping runs add risk without benefit here.♻️ Suggested fix
- concurrencyPolicy: Allow + concurrencyPolicy: ForbidAlso applies to: 141-141
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@community/CM-Configuration-Management/policy-node-image-lifecycle.yaml` at line 136, Update the concurrencyPolicy fields in the scheduled prune jobs to Forbid, including both occurrences near the referenced entries, so a slow-running job cannot overlap with the next scheduled run while retaining backoffLimit: 0.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@community/CM-Configuration-Management/policy-node-image-lifecycle.yaml`:
- Around line 139-150: Update the CronJob’s jobTemplate to fan out pruning
across every worker node rather than relying on the default single completion.
Use a per-node execution mechanism, such as a DaemonSet-compatible Job strategy
or explicit worker-node job fan-out, while preserving the existing nodeSelector
and cleanup container behavior.
- Around line 9-11: Update the nested ConfigurationPolicy.spec.remediationAction
values for policy-kubelet-image-gc, policy-image-prune-ns,
policy-image-prune-rbac, and policy-image-prune-cronjob from inform to enforce
so each resource-creating policy deploys its intended cluster objects.
- Around line 92-117: Narrow the image-gc-privileged SecurityContextConstraints
and associated CronJob so the image-gc-sa service account no longer receives
privileged host-root access through RunAsAny, hostPath volumes, or the host /
mount. Replace the chroot /host crictl workflow with access limited to the
required CRI socket and image-storage path, using read-only or minimal write
permissions, or adopt kubelet-native image garbage collection.
- Around line 61-90: Remove the image-gc-node-reader ClusterRole and
image-gc-node-reader-binding resources from the policy, while preserving the
image-gc-sa ServiceAccount and its existing SCC attachment.
- Around line 164-169: Update the image reference check in the IMG_ID pruning
loop to use a supported crictl filter, replacing the invalid --image-id option
with --image, or cross-reference containers through crictl JSON output. Preserve
the existing REF_COUNT and SKIPPED behavior so images referenced by any
container are never removed.
---
Nitpick comments:
In `@community/CM-Configuration-Management/policy-node-image-lifecycle.yaml`:
- Line 153: Replace the :latest tag in the policy-node-image-lifecycle image
reference with a specific, immutable ose-cli version tag, preserving the
existing registry and repository while making pruning job runs reproducible.
- Line 136: Update the concurrencyPolicy fields in the scheduled prune jobs to
Forbid, including both occurrences near the referenced entries, so a
slow-running job cannot overlap with the next scheduled run while retaining
backoffLimit: 0.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: f4759401-a299-4ebd-8817-3443186ab089
📒 Files selected for processing (2)
community/CM-Configuration-Management/policy-node-image-lifecycle.yamlcommunity/README.md
- Fix crictl ps flag: --image-id -> --image - Fix timestamp parsing to use info.imageSpec.created with fromisoformat() - Use host python3 via chroot for consistent parsing - Add referenced/removed/skipped counters Signed-off-by: Dustin Trapani <12038788+DustinTrap@users.noreply.github.com> Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Dustin Trapani <12038788+DustinTrap@users.noreply.github.com>
CronJob with nodeSelector only ran on one random worker per week. MachineConfig delivers the prune script to every worker node via the MachineConfigPool, matching how the KubeletConfig already targets all workers. Eliminates Namespace, ServiceAccount, RBAC, and custom SCC. Script runs natively on nodes as a systemd oneshot timer (weekly, Sunday 2am). Signed-off-by: Dustin Trapani <12038788+DustinTrap@users.noreply.github.com> Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Dustin Trapani <12038788+DustinTrap@users.noreply.github.com>
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
community/CM-Configuration-Management/policy-node-image-lifecycle.yaml (2)
59-62: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winEmbedding the prune script as base64 makes it unreviewable and undiffable.
Ignition accepts plaintext data URLs (
data:text/plain;charset=utf-8,<percent-encoded>), which keeps the script visible in review and ingit diff. Alternatively, keep a canonicalimage-gc-prune.shin the repo plus a small generator so the encoded blob is reproducible.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@community/CM-Configuration-Management/policy-node-image-lifecycle.yaml` around lines 59 - 62, Replace the base64-encoded contents of image-gc-prune.sh in the policy-node-image-lifecycle configuration with an Ignition plaintext data URL using percent-encoded script content, keeping the prune script directly visible and diffable. Preserve the existing script behavior and executable mode; alternatively, use a canonical repository script with a reproducible generator for the encoded data.
47-52: 🩺 Stability & Availability | 🔵 TrivialDocument the rolling worker reboot this policy triggers.
Applying
99-worker-image-gc-prunemakes MCO drain and reboot every worker node, and theKubeletConfigin the first template triggers a second rollout of the same pool. On large fleets that is a long, disruptive window. Worth noting in the policy header comment / README entry so consumers schedule it in a maintenance window, and consider whether both templates can land in one rollout.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@community/CM-Configuration-Management/policy-node-image-lifecycle.yaml` around lines 47 - 52, Document in the policy header comment or README entry that applying 99-worker-image-gc-prune causes MCO to drain and reboot every worker, with the KubeletConfig triggering a second worker-pool rollout; advise scheduling it during a maintenance window and assess whether both templates can be combined into a single rollout.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@community/CM-Configuration-Management/policy-node-image-lifecycle.yaml`:
- Line 62: Replace both python3-based JSON parsing pipelines in the embedded
pruning script with the available jq-based extraction, preserving container
image reference and creation timestamp retrieval. Validate that the parser is
available before processing and emit a distinct logger failure when it is
missing or parsing fails; do not silently continue with empty values or
counters.
- Line 62: Update the embedded pruning script to normalize container imageRef
values and crictl image IDs to the same sha256 digest format before the
CONTAINER_IMAGES reference check. In the pruning loop, increment REMOVED only
when crictl rmi succeeds; otherwise increment a local FAILED counter. Include
FAILED in the final logger summary alongside the existing counters.
---
Nitpick comments:
In `@community/CM-Configuration-Management/policy-node-image-lifecycle.yaml`:
- Around line 59-62: Replace the base64-encoded contents of image-gc-prune.sh in
the policy-node-image-lifecycle configuration with an Ignition plaintext data
URL using percent-encoded script content, keeping the prune script directly
visible and diffable. Preserve the existing script behavior and executable mode;
alternatively, use a canonical repository script with a reproducible generator
for the encoded data.
- Around line 47-52: Document in the policy header comment or README entry that
applying 99-worker-image-gc-prune causes MCO to drain and reboot every worker,
with the KubeletConfig triggering a second worker-pool rollout; advise
scheduling it during a maintenance window and assess whether both templates can
be combined into a single rollout.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 5b00bda0-b72c-4bdd-90a2-000837e0e14d
📒 Files selected for processing (1)
community/CM-Configuration-Management/policy-node-image-lifecycle.yaml
- Replace python3 with jq (RHCOS ships jq, not /usr/bin/python3) - Normalize image IDs for consistent matching - Track failed removals separately Signed-off-by: Dustin Trapani <12038788+DustinTrap@users.noreply.github.com> Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Dustin Trapani <12038788+DustinTrap@users.noreply.github.com>
Contributor
|
@DustinTrap tnx a lot for this PR, we will review, on a first look I wonder if the Policy can be templatized a bit more we will come back! |
Author
|
@ch-stark I'll take a pass at templatizing what I can. Might get halfway to where you want it. Stay tuned. |
Per reviewer feedback, use hub templates (fromConfigMap) to make GC thresholds configurable per managed cluster. The policy now: - Creates a ConfigMap (image-gc-config) with default values on each managed cluster's namespace - Reads imageGCHighThresholdPercent, imageGCLowThresholdPercent, and imageMinimumGCAge from that ConfigMap via hub templates - Allows per-cluster customization by editing the ConfigMap values before or after policy enforcement The MachineConfig prune script's MAX_AGE_DAYS remains in the base64 blob (documented in header comments) since MachineConfig file contents cannot use hub templates. Signed-off-by: Dustin Trapani <12038788+DustinTrap@users.noreply.github.com> Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Dustin Trapani <12038788+DustinTrap@users.noreply.github.com>
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@community/CM-Configuration-Management/policy-node-image-lifecycle.yaml`:
- Line 33: Update the top-level Policy remediationAction to enforce for this
resource-creating policy, ensuring the ConfigMap, KubeletConfig, and
MachineConfig templates can be created; leave inform only on any check-only
policies or templates.
- Around line 1-14: Remove the embedded image-gc-config provisioning from
policy-image-gc-config and update the prerequisite comment to state that the
ConfigMap must be managed separately on the hub. Preserve the existing hub-side
hub fromConfigMap .ManagedClusterName consumers, unless intentionally switching
the entire consumer flow to managed-cluster templating.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 72bdd42f-47b8-43f6-b830-fbc621556765
📒 Files selected for processing (1)
community/CM-Configuration-Management/policy-node-image-lifecycle.yaml
The policy-image-gc-config template created the ConfigMap on the managed cluster, but hub fromConfigMap reads from the hub cluster's namespace. Remove the contradictory template and clarify the prerequisite: the ConfigMap must be created on the hub in each managed cluster's namespace before enforcing. Document why remediationAction remains inform: enforcing triggers an MCP rollout that reboots worker nodes, so users should review values and schedule a maintenance window first. Signed-off-by: Dustin Trapani <12038788+DustinTrap@users.noreply.github.com> Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Dustin Trapani <12038788+DustinTrap@users.noreply.github.com>
JustinKuli
requested changes
Aug 4, 2026
JustinKuli
left a comment
JustinKuli
left a comment
Member
There was a problem hiding this comment.
Looks good overall! I would just like to document that script somehow, instead of it just being a base64 encoded blob.
| - path: /usr/local/bin/image-gc-prune.sh | ||
| mode: 0755 | ||
| contents: | ||
| source: data:text/plain;charset=utf-8;base64,IyEvYmluL2Jhc2gKIyBQcnVuZSB1bnJlZmVyZW5jZWQgY29udGFpbmVyIGltYWdlcyBvbGRlciB0aGFuIE1BWF9BR0VfREFZUwpNQVhfQUdFX0RBWVM9NwpOT1c9JChkYXRlICslcykKQ1VUT0ZGPSQoKE5PVyAtIE1BWF9BR0VfREFZUyAqIDg2NDAwKSkKUkVNT1ZFRD0wClNLSVBQRUQ9MApSRUZFUkVOQ0VEPTAKVU5QQVJTQUJMRT0wCkZBSUxFRD0wCgojIEJ1aWxkIHNldCBvZiBpbWFnZSBJRHMgcmVmZXJlbmNlZCBieSBhbnkgY29udGFpbmVyIChvbmUgY2FsbCwgbm90IE4pCkNPTlRBSU5FUl9JTUFHRVM9JChjcmljdGwgcHMgLWEgLW8ganNvbiAyPi9kZXYvbnVsbCBcCiAgfCBqcSAtciAnLmNvbnRhaW5lcnNbXT8gfCAuaW1hZ2VSZWYgLy8gZW1wdHknIDI+L2Rldi9udWxsIFwKICB8IHNlZCAnc3wuKkB8fDsgc3xec2hhMjU2Onx8JyB8IHNvcnQgLXUpCgpmb3IgSU1HX0lEIGluICQoY3JpY3RsIGltYWdlcyAtcSk7IGRvCiAgU0hPUlRfSUQ9JChlY2hvICIkSU1HX0lEIiB8IHNlZCAnc3xec2hhMjU2Onx8JykKICBpZiBlY2hvICIkQ09OVEFJTkVSX0lNQUdFUyIgfCBncmVwIC1xRiAiJFNIT1JUX0lEIjsgdGhlbgogICAgUkVGRVJFTkNFRD0kKChSRUZFUkVOQ0VEICsgMSkpCiAgICBjb250aW51ZQogIGZpCgogICMgRXh0cmFjdCBjcmVhdGlvbiB0aW1lc3RhbXAgKGpxIGZvciBKU09OLCBkYXRlIC1kIGZvciBwYXJzaW5nKQogIFRTPSQoY3JpY3RsIGluc3BlY3RpICIkSU1HX0lEIiAyPi9kZXYvbnVsbCBcCiAgICB8IGpxIC1yICcoLmluZm8uaW1hZ2VTcGVjLmNyZWF0ZWQgLy8gLnN0YXR1cy5jcmVhdGVkQXQgLy8gZW1wdHkpJyAyPi9kZXYvbnVsbCkKICBDUkVBVEVEX1RTPSQoZGF0ZSAtZCAiJFRTIiArJXMgMj4vZGV2L251bGwgfHwgZWNobyAwKQoKICBpZiBbICIkQ1JFQVRFRF9UUyIgLWVxIDAgXTsgdGhlbgogICAgVU5QQVJTQUJMRT0kKChVTlBBUlNBQkxFICsgMSkpCiAgICBjb250aW51ZQogIGZpCgogIGlmIFsgIiRDUkVBVEVEX1RTIiAtbHQgIiRDVVRPRkYiIF07IHRoZW4KICAgIGlmIGNyaWN0bCBybWkgIiRJTUdfSUQiIDI+JjE7IHRoZW4KICAgICAgUkVNT1ZFRD0kKChSRU1PVkVEICsgMSkpCiAgICBlbHNlCiAgICAgIEZBSUxFRD0kKChGQUlMRUQgKyAxKSkKICAgIGZpCiAgZWxzZQogICAgU0tJUFBFRD0kKChTS0lQUEVEICsgMSkpCiAgZmkKZG9uZQoKbG9nZ2VyIC10IGltYWdlLWdjLXBydW5lICJyZWZlcmVuY2VkPSRSRUZFUkVOQ0VEIHJlbW92ZWQ9JFJFTU9WRUQgc2tpcHBlZD0kU0tJUFBFRCB1bnBhcnNhYmxlPSRVTlBBUlNBQkxFIGZhaWxlZD0kRkFJTEVEIgo= |
Member
There was a problem hiding this comment.
It should be possible to do the base64 encoding in the policy engine, although it takes a few tricks with templates, and some changing of quote types. Basically, the construction is {{- list <code lines> "" | join "\n" | base64enc -}}
Honestly, after wrangling this together, I'm not sure if it's worth it... But at the very least, the script should be documented in a comment here in the collection, and if it's been copied from somewhere else, a link back to the source.
Suggested change
| source: data:text/plain;charset=utf-8;base64,IyEvYmluL2Jhc2gKIyBQcnVuZSB1bnJlZmVyZW5jZWQgY29udGFpbmVyIGltYWdlcyBvbGRlciB0aGFuIE1BWF9BR0VfREFZUwpNQVhfQUdFX0RBWVM9NwpOT1c9JChkYXRlICslcykKQ1VUT0ZGPSQoKE5PVyAtIE1BWF9BR0VfREFZUyAqIDg2NDAwKSkKUkVNT1ZFRD0wClNLSVBQRUQ9MApSRUZFUkVOQ0VEPTAKVU5QQVJTQUJMRT0wCkZBSUxFRD0wCgojIEJ1aWxkIHNldCBvZiBpbWFnZSBJRHMgcmVmZXJlbmNlZCBieSBhbnkgY29udGFpbmVyIChvbmUgY2FsbCwgbm90IE4pCkNPTlRBSU5FUl9JTUFHRVM9JChjcmljdGwgcHMgLWEgLW8ganNvbiAyPi9kZXYvbnVsbCBcCiAgfCBqcSAtciAnLmNvbnRhaW5lcnNbXT8gfCAuaW1hZ2VSZWYgLy8gZW1wdHknIDI+L2Rldi9udWxsIFwKICB8IHNlZCAnc3wuKkB8fDsgc3xec2hhMjU2Onx8JyB8IHNvcnQgLXUpCgpmb3IgSU1HX0lEIGluICQoY3JpY3RsIGltYWdlcyAtcSk7IGRvCiAgU0hPUlRfSUQ9JChlY2hvICIkSU1HX0lEIiB8IHNlZCAnc3xec2hhMjU2Onx8JykKICBpZiBlY2hvICIkQ09OVEFJTkVSX0lNQUdFUyIgfCBncmVwIC1xRiAiJFNIT1JUX0lEIjsgdGhlbgogICAgUkVGRVJFTkNFRD0kKChSRUZFUkVOQ0VEICsgMSkpCiAgICBjb250aW51ZQogIGZpCgogICMgRXh0cmFjdCBjcmVhdGlvbiB0aW1lc3RhbXAgKGpxIGZvciBKU09OLCBkYXRlIC1kIGZvciBwYXJzaW5nKQogIFRTPSQoY3JpY3RsIGluc3BlY3RpICIkSU1HX0lEIiAyPi9kZXYvbnVsbCBcCiAgICB8IGpxIC1yICcoLmluZm8uaW1hZ2VTcGVjLmNyZWF0ZWQgLy8gLnN0YXR1cy5jcmVhdGVkQXQgLy8gZW1wdHkpJyAyPi9kZXYvbnVsbCkKICBDUkVBVEVEX1RTPSQoZGF0ZSAtZCAiJFRTIiArJXMgMj4vZGV2L251bGwgfHwgZWNobyAwKQoKICBpZiBbICIkQ1JFQVRFRF9UUyIgLWVxIDAgXTsgdGhlbgogICAgVU5QQVJTQUJMRT0kKChVTlBBUlNBQkxFICsgMSkpCiAgICBjb250aW51ZQogIGZpCgogIGlmIFsgIiRDUkVBVEVEX1RTIiAtbHQgIiRDVVRPRkYiIF07IHRoZW4KICAgIGlmIGNyaWN0bCBybWkgIiRJTUdfSUQiIDI+JjE7IHRoZW4KICAgICAgUkVNT1ZFRD0kKChSRU1PVkVEICsgMSkpCiAgICBlbHNlCiAgICAgIEZBSUxFRD0kKChGQUlMRUQgKyAxKSkKICAgIGZpCiAgZWxzZQogICAgU0tJUFBFRD0kKChTS0lQUEVEICsgMSkpCiAgZmkKZG9uZQoKbG9nZ2VyIC10IGltYWdlLWdjLXBydW5lICJyZWZlcmVuY2VkPSRSRUZFUkVOQ0VEIHJlbW92ZWQ9JFJFTU9WRUQgc2tpcHBlZD0kU0tJUFBFRCB1bnBhcnNhYmxlPSRVTlBBUlNBQkxFIGZhaWxlZD0kRkFJTEVEIgo= | |
| source: 'data:text/plain;charset=utf-8;base64,{{- list | |
| "#!/bin/bash" | |
| "# Prune unreferenced container images older than MAX_AGE_DAYS" | |
| "MAX_AGE_DAYS=7" | |
| "NOW=$(date +%s)" | |
| "CUTOFF=$((NOW - MAX_AGE_DAYS * 86400))" | |
| "REMOVED=0" | |
| "SKIPPED=0" | |
| "REFERENCED=0" | |
| "UNPARSABLE=0" | |
| "FAILED=0" | |
| "" | |
| "# Build set of image IDs referenced by any container (one call, not N)" | |
| "CONTAINER_IMAGES=$(crictl ps -a -o json 2>/dev/null \\" | |
| " | jq -r \".containers[]? | .imageRef // empty\" 2>/dev/null \\" | |
| " | sed \"s|.*@||; s|^sha256:||\" | sort -u)" | |
| "" | |
| "for IMG_ID in $(crictl images -q); do" | |
| " SHORT_ID=$(echo \"$IMG_ID\" | sed \"s|^sha256:||\")" | |
| " if echo \"$CONTAINER_IMAGES\" | grep -qF \"$SHORT_ID\"; then" | |
| " REFERENCED=$((REFERENCED + 1))" | |
| " continue" | |
| " fi" | |
| "" | |
| " # Extract creation timestamp (jq for JSON, date -d for parsing)" | |
| " TS=$(crictl inspecti \"$IMG_ID\" 2>/dev/null \\" | |
| " | jq -r \"(.info.imageSpec.created // .status.createdAt // empty)\" 2>/dev/null)" | |
| " CREATED_TS=$(date -d \"$TS\" +%s 2>/dev/null || echo 0)" | |
| "" | |
| " if [ \"$CREATED_TS\" -eq 0 ]; then" | |
| " UNPARSABLE=$((UNPARSABLE + 1))" | |
| " continue" | |
| " fi" | |
| "" | |
| " if [ \"$CREATED_TS\" -lt \"$CUTOFF\" ]; then" | |
| " if crictl rmi \"$IMG_ID\" 2>&1; then" | |
| " REMOVED=$((REMOVED + 1))" | |
| " else" | |
| " FAILED=$((FAILED + 1))" | |
| " fi" | |
| " else" | |
| " SKIPPED=$((SKIPPED + 1))" | |
| " fi" | |
| "done" | |
| "" | |
| "logger -t image-gc-prune \"referenced=$REFERENCED removed=$REMOVED skipped=$SKIPPED unparsable=$UNPARSABLE failed=$FAILED\"" | |
| "" | |
| | join "\n" | base64enc -}}' |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
policy-node-image-lifecycleunderCM-Configuration-ManagementProblem
OpenShift's kubelet image garbage collection is purely disk-pressure-driven (default thresholds: 85%/80%). On nodes with large disks, old image versions from prior OCP releases accumulate indefinitely. Image-based security scanners inspect all cached images — not just running ones — leading to false vulnerability reports on already-patched containers.
Solution
imageGCHighThresholdPercent: 65,imageGCLowThresholdPercent: 50,imageMinimumGCAge: 5m) — triggers GC earlier and cleans more aggressivelyValidation
Tested on OCP 4.22.7 with ACM 2.17.0:
kubelet.conf)Compatibility
batch/v1CronJob)ConfigurationPolicyv1)NIST SP 800-53 Mapping
/cc @grimsleydl
🤖 Generated with Claude Code
Summary by CodeRabbit