Feature Request: Allow configuration of resources for the `shutdown-manager` container

[davinkevin]

Hello Contour team 👋

What happened?

In our Kubernetes environment, we are observing CPU throttling on the shutdown-manager container within the Envoy DaemonSet.

What you expected to happen?

We expected to be able to configure the CPU/memory requests and limits for the shutdown-manager container, in the same way that resources can be configured for the contour and envoy containers.

How to reproduce it?

This issue is noticeable in clusters under load where the shutdown-manager requires more CPU than the default allocation to perform its duties, leading to throttling by the Kubernetes scheduler.

The core of the issue is that there appears to be no API field to define these resources.

Current Environment

• Contour version: v1.33.1
• Installation method: Contour Operator using the ContourDeployment CRD with GatewayAPI.
• Kubernetes version: 1.34.x
• Cloud provider or hardware: Self-hosted
• Orchestration: GitOps-based with FluxCD

Proposed Solution

To solve this, we propose adding a new field to the ContourDeployment API specification. A logical location would be under spec.envoy, mirroring the structure of other components.

Example of the desired configuration in the ContourDeployment resource:

apiVersion: projectcontour.io/v1alpha1
kind: ContourDeployment
metadata:
  name: internal
  namespace: projectcontour
spec:
  # ... other contour/envoy settings
  envoy:
    # ... other envoy settings
    shutdownManager:
      resources:
        requests:
          cpu: 100m
        limits:
          cpu: 200m
    # ... other envoy settings

Workarounds Considered

We are using the Contour Operator in a GitOps workflow, which means the Envoy DaemonSet is created and reconciled at runtime. This prevents us from using a standard Kustomize patch in our Git repository to modify the DaemonSet, as the Operator would likely overwrite any manual changes during reconciliation.

Furthermore, we've identified that the containers resource limits are currently hardcoded in the Contour codebase

Without a way to specify these values through the ContourDeployment CRD, we have no clear path to resolving the CPU throttling issue. If you have an idea, it's welcome in the meantime.

Thank you for considering this feature.

[tsaarni]

Hi @davinkevin, I see and it makes sense, but I wonder if it would be enough to just increase the hard-coded CPU limit in the code significantly? The logic of the shutdown-manager is very simple: it waits for the pod shutdown sequence to start and then polls Envoy for a short time before pod is deleted. It seems unlikely to me that shutdown-manager would cause any issues even it was given overly generous CPU limit. It could avoid throttling without requiring each user to do unnecessary dimensioning and configuration.

[davinkevin]

Hello @tsaarni and thank you for your answer.

I'm totally open to this idea, but we know CPU may differ (power, arch…) and some could be subject to this kind of issue while others are not.

Or the solution could be a conjunction of your point and mine at the same time. A default value increased and a potential configuration change "just in case".
Because, to be honest, the creation of a rule to silence the envoy component in charge of all our traffic is quite concerning as a mistake or misconfiguration could silence any issue encountered by pods supporting all production flow. That to say a mitigation where we can increase CPU is IMO a safer fallback.

[davinkevin]

👋

If you have any question, I'm able to answer. For the moment, we mitigate the situation but it's quite complex to silence contour all the time… and force us to keep alertmanager state between restarts 😅.

Is there, at least, an increased plan as part of the next patch version for 1.33.2 maybe?

Additionally, I see a lot of unexpected restart of the contour component, for a reason I ignore:

And CPU Throttling is pro-eminent now (view: last 30 days):

Around this date, we upgraded (VM backup + restore, no change at host level) our virtualization system, but nothing else.

[tsaarni]

Hi @davinkevin I don't have time to make the values configurable, but I submitted #7382. Can you please review it and let me know if you think it helps or if it should be changed?

[davinkevin]

LGTM. I made a proper review but my role is just for information, no more. 😇

[tsaarni]

I've cherry-picked the patch for release-1.33 maintenance branch #7407.

[tsaarni]

This is included in https://github.com/projectcontour/contour/releases/tag/v1.33.2

[davinkevin]

Merged!

The current state for the last 60days was like this:

I created a reminder in 7d so I can report if the problem is soled. Let's see in a week 😇

[davinkevin]

To be honest, that isn't very better if we compare dashboards:

Do you know what could be the issue here? 🤔

[tsaarni]

I am not sure, but the shutdown manager should remain inactive until the shutdown procedure is started. Which specific metrics are collected and how exactly it the throttling graph calculated?