Skip to content

fix(exporter): forward Authorization header from zxp to zot metrics endpoint#4019

Open
AkashKumar7902 wants to merge 1 commit into
project-zot:mainfrom
AkashKumar7902:issue-3263-zxp-forward-auth-header
Open

fix(exporter): forward Authorization header from zxp to zot metrics endpoint#4019
AkashKumar7902 wants to merge 1 commit into
project-zot:mainfrom
AkashKumar7902:issue-3263-zxp-forward-auth-header

Conversation

@AkashKumar7902

Copy link
Copy Markdown
Contributor

What type of PR is this?

bug

Which issue does this PR fix:

Fixes #3263

What does this PR do / Why do we need it:

When zot is configured with authentication (e.g. htpasswd), the zxp node exporter could not scrape zot's /metrics endpoint because it always issued an unauthenticated request. The JSON decoder silently consumed the 401 error body, so the resulting Prometheus output was missing every zot_* metric (only go_*/process_* showed up) and zot_up never flipped to 0 to surface the failure.

This PR makes the exporter forward the Authorization header from the inbound /metrics request to the upstream zot scrape:

  • pkg/exporter/api/exporter.go swaps the global prometheus.Register + promhttp.Handler() wiring for a per-request handler. Each request now builds a fresh prometheus.Registry and registers a collector clone scoped to that request via Collector.WithHeaders, which carries the forwarded Authorization header(s) into the scrape.
  • pkg/extensions/monitoring/minimal_client.go adds MetricsClient.WithHeaders and copies any configured headers onto every outgoing request in makeGETRequest. Non-2xx responses are now treated as scrape failures instead of being decoded as JSON, so zot_up correctly reports 0 and the underlying error is logged.

Behavior with no auth on zot is unchanged because no Authorization header is forwarded when none is sent.

If an issue # is not available please add repro steps and logs showing the issue:

N/A — see #3263 for repro steps and logs.

Testing done on this change:

  • Added TestMetricsHandlerForwardsAuthorizationHeader and TestMetricsHandlerReportsZotDownWithoutAuthorizationHeader in pkg/exporter/api/exporter_internal_test.go which spin up an httptest server that requires a Basic Authorization header. The first test asserts the exporter forwards the header (200 + zot_up 1); the second asserts that without the header the scrape is treated as a failure (zot_up 0).
  • Added TestMetricsClientForwardsHeadersAndRejectsNonSuccessStatus in pkg/extensions/monitoring/minimal_client_test.go which verifies that an unauthenticated GetMetrics() call now returns unexpected status code 401 (instead of decoding the error body) and that calling WithHeaders produces a client whose request carries the Authorization header.
  • go build ./pkg/exporter/... ./pkg/extensions/monitoring/...
  • go test ./pkg/exporter/api/... ./pkg/extensions/monitoring/... — both packages pass locally.
  • gofmt -l and go vet clean on the touched files.

Automation added to e2e:

No e2e change. The fix is fully covered by the new unit/internal tests against an httptest server that mirrors a 401-protected zot. An e2e test that boots a real htpasswd-protected zot under zxp can be tracked separately if desired.

Will this break upgrades or downgrades?

No. The wire format of the /metrics endpoint is unchanged. Existing zxp deployments that do not pass an Authorization header continue to work exactly as before; the only behavior change is that scrape failures (any non-2xx from zot) now correctly surface as zot_up 0 rather than an empty success.

Does this PR introduce any user-facing change?:

Yes — operators running zxp against a zot with authentication enabled can now scrape it by forwarding their Prometheus credentials (e.g. basic_auth in the Prometheus scrape config pointed at zxp). zxp passes the inbound Authorization header straight through to zot.

fix(exporter): zxp now forwards the inbound `Authorization` header to zot's /metrics endpoint, so it can scrape zot when authentication (e.g. htpasswd) is enabled. Non-2xx responses from zot are also treated as scrape failures and reported via `zot_up 0`.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…ndpoint

zxp scrapes zot's /metrics endpoint to translate the JSON payload into
Prometheus format, but until now it always sent an unauthenticated
request. With htpasswd or any other authentication enabled on zot, the
scrape returned 401 and the JSON decoder silently produced an empty
MetricsInfo, so zxp only emitted go_*/process_* metrics and the
zot_up gauge was never set to 0 to surface the failure.

This change:

- Replaces the global prometheus.Register + promhttp.Handler() wiring
  in zxp with a per-request handler that builds a fresh registry,
  copies the inbound Authorization header(s), and registers a
  collector clone scoped to that request via Collector.WithHeaders.
- Adds MetricsClient.WithHeaders so the underlying scrape carries the
  forwarded credentials to zot, and applies any configured headers on
  every outgoing request in makeGETRequest.
- Treats non-2xx responses from zot as scrape failures so zot_up flips
  to 0 (and the error is logged) instead of silently decoding an HTTP
  error body as metrics.

Fixes project-zot#3263

Signed-off-by: Akash Kumar <meakash7902@gmail.com>
@rchincha

rchincha commented May 1, 2026

Copy link
Copy Markdown
Contributor

@AkashKumar7902 do you plan to work on these PRs? Some of them have been moved out of draft state (ones we like).

@rchincha rchincha added this to the v2.1.17 milestone May 1, 2026
@AkashKumar7902

Copy link
Copy Markdown
Contributor Author

@rchincha yes

@rchincha

rchincha commented May 8, 2026

Copy link
Copy Markdown
Contributor

@AkashKumar7902 pls fix the merge conflicts

@codecov

codecov Bot commented May 8, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.32258% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.44%. Comparing base (8282aef) to head (281a7eb).
⚠️ Report is 47 commits behind head on main.

Files with missing lines Patch % Lines
pkg/exporter/api/exporter.go 86.36% 2 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4019      +/-   ##
==========================================
+ Coverage   91.42%   91.44%   +0.01%     
==========================================
  Files         197      197              
  Lines       28152    28179      +27     
==========================================
+ Hits        25739    25768      +29     
+ Misses       1562     1560       -2     
  Partials      851      851              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@rchincha rchincha modified the milestones: v2.1.17, v2.1.18 May 12, 2026
@rchincha rchincha modified the milestones: v2.1.18, v2.1.19 Jun 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: zxp doesn't export zot metrics with turned on authentication

2 participants