What is the recommended way to have metrics enabled in Argo Workflows?

[milarze]

I have kube-prometheus-stack installed and I enabled metrics in the values file:

controller:
  metricsConfig:
    enabled: true
  serviceMonitor:
    enabled: true

Taking a look at the service monitor, I see it is scraping over HTTPS:

spec:
  endpoints:
  - honorLabels: false
    interval: 30s
    path: /metrics
    port: metrics
    scheme: https

And looking at the Service:

  ports:
  - name: metrics
    port: 8080
    protocol: TCP
    targetPort: 9090

I tried port forwarding and using curl -k and it is serving on 8080 over HTTPS.

Checking in the Prometheus UI I see the target is down and the following error:

Error scraping target: Get "https://10.0.6.42:9090/metrics": tls: failed to verify certificate: x509: cannot validate certificate for 10.0.6.42 because it doesn't contain any IP SANs

So the service is using a self-signed cert and it cannot be properly verified. What is the recommended way to resolve this?

Chart Version: 0.45.14
AppVersion: 3.6.7

[milarze]

I've setup without HTTPS by setting secure: false.

controller:
  metricsConfig:
    enabled: true
    secure: false
  serviceMonitor:
    enabled: true

I did not figure out how to make it work over HTTPS.