Skip to content

Commit d95ec71

Browse files
bert-ebert-e
committed
Merge branch 'w/134.0/bugfix/bump-thanos-limit' into tmp/octopus/q/134.0
2 parents 5f15fcc + 8021950 commit d95ec71

10 files changed

Lines changed: 156 additions & 7 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,19 @@
2323

2424
## Release 133.0.11 (in development)
2525

26+
### Bug Fixes
27+
28+
- Avoid the Thanos Querier being OOMKilled on heavy queries (such as the
29+
`sosreport` metrics collection): its CPU/memory requests and limits are now
30+
configurable through the `metalk8s-thanos-config` Cluster and Service
31+
ConfigMap, and the default memory limit is raised from 192Mi to 2Gi. The
32+
Querier also now runs with two replicas for high availability.
33+
(PR[#5008](https://github.com/scality/metalk8s/pull/5008))
34+
35+
- Use the `EndpointSlice` service discovery role for Prometheus to stop the
36+
deprecated `v1 Endpoints` API warnings logged on Kubernetes 1.33+
37+
(PR[#5008](https://github.com/scality/metalk8s/pull/5008))
38+
2639
## Release 133.0.10
2740

2841
### Enhancements

‎buildchain/buildchain/codegen.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -277,6 +277,9 @@ def codegen_chart_thanos() -> types.TaskDict:
277277
cmd = (
278278
f"{constants.CHART_RENDER_CMD} thanos {value_file} {chart_dir} "
279279
"--namespace metalk8s-monitoring "
280+
"--service-config thanos metalk8s-thanos-config "
281+
"metalk8s/addons/prometheus-operator/config/thanos.yaml "
282+
"metalk8s-monitoring "
280283
f"--output {target_sls}"
281284
)
282285

‎buildchain/buildchain/salt_tree.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -388,6 +388,7 @@ def _download_ui_operator_crds() -> str:
388388
Path("salt/metalk8s/addons/prometheus-operator/config/alertmanager.yaml"),
389389
Path("salt/metalk8s/addons/prometheus-operator/config/grafana.yaml.j2"),
390390
Path("salt/metalk8s/addons/prometheus-operator/config/prometheus.yaml"),
391+
Path("salt/metalk8s/addons/prometheus-operator/config/thanos.yaml"),
391392
Path(
392393
"salt/metalk8s/addons/prometheus-operator/deployed/",
393394
"alertmanager-configuration-secret.sls",

‎charts/kube-prometheus-stack.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,10 @@ prometheus:
106106
enabled: true
107107

108108
prometheusSpec:
109+
# Enforce EndpointSlice usage since default "Endpoints" are deprecated
110+
# and cause warning in the prometheus logs
111+
serviceDiscoveryRole: EndpointSlice
112+
109113
thanos:
110114
image: '__full_image__(thanos)'
111115
objectStorageConfig: {}

‎charts/thanos.yaml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,3 +84,16 @@ query:
8484

8585
nodeSelector:
8686
node-role.kubernetes.io/infra: ''
87+
88+
replicaCount: 2
89+
90+
# cpu/memory templated from CSC (metalk8s-thanos-config) so limits stay tunable
91+
resources:
92+
requests:
93+
cpu: '__var__(thanos.spec.deployment.resources.requests.cpu)'
94+
memory: '__var__(thanos.spec.deployment.resources.requests.memory)'
95+
ephemeral-storage: 50Mi
96+
limits:
97+
cpu: '__var__(thanos.spec.deployment.resources.limits.cpu)'
98+
memory: '__var__(thanos.spec.deployment.resources.limits.memory)'
99+
ephemeral-storage: 2Gi

‎docs/operation/cluster_and_service_configuration.rst‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,19 @@ The default configuration values for Prometheus are specified below:
7373
:lines: 3-
7474

7575

76+
Thanos Default Configuration
77+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
78+
79+
Thanos provides a global, deduplicated query view over the (highly available)
80+
Prometheus instances.
81+
82+
The default configuration values for Thanos are specified below:
83+
84+
.. literalinclude:: ../../salt/metalk8s/addons/prometheus-operator/config/thanos.yaml
85+
:language: yaml
86+
:lines: 3-
87+
88+
7689
Loki Default Configuration
7790
~~~~~~~~~~~~~~~~~~~~~~~~~~
7891

@@ -809,6 +822,61 @@ Applying configuration
809822
Any changes made to ``metalk8s-prometheus-config`` ConfigMap must then be
810823
applied with Salt.
811824

825+
.. parsed-literal::
826+
827+
root\@bootstrap $ kubectl exec --kubeconfig /etc/kubernetes/admin.conf \\
828+
-n kube-system -c salt-master salt-master-bootstrap -- \\
829+
salt-run state.sls \\
830+
metalk8s.addons.prometheus-operator.deployed \\
831+
saltenv=metalk8s-|version|
832+
833+
.. _csc-thanos-customization:
834+
835+
Thanos Configuration Customization
836+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
837+
838+
Default configuration for Thanos can be overridden by editing its
839+
Cluster and Service ConfigMap ``metalk8s-thanos-config`` in namespace
840+
``metalk8s-monitoring`` under the key ``data.config.yaml``:
841+
842+
.. code-block:: shell
843+
844+
root@bootstrap $ kubectl --kubeconfig /etc/kubernetes/admin.conf \
845+
edit configmap -n metalk8s-monitoring \
846+
metalk8s-thanos-config
847+
848+
Adjust the Thanos Querier resources
849+
"""""""""""""""""""""""""""""""""""
850+
851+
The Thanos Querier holds query results in memory, so a heavy query (e.g. the
852+
metrics collection performed by ``sosreport``) can exceed the default memory
853+
limit and get the Pod ``OOMKilled``. The CPU and memory requests and limits
854+
can be tuned:
855+
856+
.. code-block:: yaml
857+
858+
---
859+
apiVersion: v1
860+
kind: ConfigMap
861+
metadata:
862+
name: metalk8s-thanos-config
863+
namespace: metalk8s-monitoring
864+
data:
865+
config.yaml: |-
866+
apiVersion: addons.metalk8s.scality.com
867+
kind: ThanosConfig
868+
spec:
869+
deployment:
870+
resources:
871+
requests:
872+
cpu: "500m"
873+
memory: "512Mi"
874+
limits:
875+
cpu: "1"
876+
memory: "4Gi"
877+
878+
Then apply the configuration:
879+
812880
.. parsed-literal::
813881
814882
root\@bootstrap $ kubectl exec --kubeconfig /etc/kubernetes/admin.conf \\

‎salt/metalk8s/addons/prometheus-operator/config/thanos.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
#!yaml
2+
3+
# Configuration of the Thanos service
4+
apiVersion: addons.metalk8s.scality.com
5+
kind: ThanosConfig
6+
spec:
7+
# Configure the Thanos Querier Deployment
8+
deployment:
9+
resources:
10+
requests:
11+
cpu: "500m"
12+
memory: "512Mi"
13+
limits:
14+
cpu: "1"
15+
memory: "2Gi"

‎salt/metalk8s/addons/prometheus-operator/deployed/chart.sls‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80448,6 +80448,7 @@ spec:
8044880448
seccompProfile:
8044980449
type: RuntimeDefault
8045080450
serviceAccountName: prometheus-operator-prometheus
80451+
serviceDiscoveryRole: EndpointSlice
8045180452
serviceMonitorNamespaceSelector: {}
8045280453
serviceMonitorSelector:
8045380454
matchLabels:

‎salt/metalk8s/addons/prometheus-operator/deployed/service-configuration.sls‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,14 @@ include:
2525
)
2626
%}
2727

28+
{%- set thanos_config = salt.metalk8s_kubernetes.get_object(
29+
kind='ConfigMap',
30+
apiVersion='v1',
31+
namespace='metalk8s-monitoring',
32+
name='metalk8s-thanos-config',
33+
)
34+
%}
35+
2836
{%- if grafana_config is none %}
2937

3038
Create grafana-config ConfigMap:
@@ -93,3 +101,26 @@ metalk8s-alertmanager-config ConfigMap already exists:
93101
test.succeed_without_changes: []
94102

95103
{%- endif %}
104+
105+
{%- if thanos_config is none %}
106+
107+
Create thanos-config ConfigMap:
108+
metalk8s_kubernetes.object_present:
109+
- manifest:
110+
apiVersion: v1
111+
kind: ConfigMap
112+
metadata:
113+
name: metalk8s-thanos-config
114+
namespace: metalk8s-monitoring
115+
data:
116+
config.yaml: |-
117+
apiVersion: addons.metalk8s.scality.com
118+
kind: ThanosConfig
119+
spec: {}
120+
121+
{%- else %}
122+
123+
metalk8s-thanos-config ConfigMap already exists:
124+
test.succeed_without_changes: []
125+
126+
{%- endif %}

‎salt/metalk8s/addons/prometheus-operator/deployed/thanos-chart.sls‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,8 @@
22

33
{%- from "metalk8s/map.jinja" import repo with context %}
44
{%- from "metalk8s/repo/macro.sls" import build_image_name with context %}
5-
6-
5+
{%- set thanos_defaults = salt.slsutil.renderer('salt://metalk8s/addons/prometheus-operator/config/thanos.yaml', saltenv=saltenv) %}
6+
{%- set thanos = salt.metalk8s_service_configuration.get_service_conf('metalk8s-monitoring', 'metalk8s-thanos-config', thanos_defaults) %}
77
88
{% raw %}
99
@@ -124,7 +124,7 @@ metadata:
124124
name: thanos-query
125125
namespace: metalk8s-monitoring
126126
spec:
127-
replicas: 1
127+
replicas: 2
128128
revisionHistoryLimit: 10
129129
selector:
130130
matchLabels:
@@ -203,13 +203,13 @@ spec:
203203
timeoutSeconds: 30
204204
resources:
205205
limits:
206-
cpu: 150m
206+
cpu: {% endraw -%}{{ thanos.spec.deployment.resources.limits.cpu }}{%- raw %}
207207
ephemeral-storage: 2Gi
208-
memory: 192Mi
208+
memory: {% endraw -%}{{ thanos.spec.deployment.resources.limits.memory }}{%- raw %}
209209
requests:
210-
cpu: 100m
210+
cpu: {% endraw -%}{{ thanos.spec.deployment.resources.requests.cpu }}{%- raw %}
211211
ephemeral-storage: 50Mi
212-
memory: 128Mi
212+
memory: {% endraw -%}{{ thanos.spec.deployment.resources.requests.memory }}{%- raw %}
213213
securityContext:
214214
allowPrivilegeEscalation: false
215215
capabilities:

0 commit comments

Comments
 (0)