Skip to content

Fix StringSliceCSV treating explicit empty string as a one-element list - #7714

Merged
SungJin1212 merged 3 commits into
cortexproject:masterfrom
pujitha24:auto/issue-6581
Sep 8, 2026
Merged

Fix StringSliceCSV treating explicit empty string as a one-element list#7714
SungJin1212 merged 3 commits into
cortexproject:masterfrom
pujitha24:auto/issue-6581

Conversation

@pujitha24

Copy link
Copy Markdown
Contributor

Motivation:
Issue #6581 reports that setting compactor.enabled_tenants: ""
explicitly in config causes the compactor to skip every tenant,
instead of behaving like the unset/omitted case (all tenants
allowed). The compactor logs "skipping user because it is not owned
by this shard" for every real tenant.

Approach:
StringSliceCSV.Set (pkg/util/flagext/stringslicecsv.go), which
backs -compactor.enabled-tenants/-compactor.disabled-tenants and
several other CSV-list flags (alertmanager, ruler, store-gateway,
ingester, querier, distributor, ring, limits, etc.), implemented
Set as strings.Split(s, ","). Go's strings.Split("", ",")
returns [""], a one-element slice containing an empty string, not
an empty/nil slice. YAML unmarshaling for enabled_tenants: ""
calls Set(""), so EnabledTenants ends up as [""] with
len() == 1.

users.NewAllowedTenants (pkg/util/users/allowed_tenants.go) treats
len(enabled) > 0 as "only tenants in this list are allowed", so
with enabled == [""] only a tenant literally named "" is
allowed and every real tenant is rejected. This same
NewAllowedTenants helper backs the identical enabled/disabled
tenant list pattern in the compactor, alertmanager, ruler, and
store-gateway, so all four are affected identically by explicitly
empty tenant-list config, though issue #6581 only reports it for the
compactor.

Fix Set to special-case an empty input string by setting the value
to nil, matching the sibling SecretStringSliceCSV.Set, which
already special-cases s == "" this same way.

Validation:
go build ./...
go test ./pkg/util/flagext/... ./pkg/compactor/... ./pkg/util/validation/... ./pkg/util/users/... -tags "netgo slicelabels"
All pass, including new tests in pkg/util/flagext/stringslicecsv_test.go
covering direct Set("") and YAML unmarshaling of csv: "".

Before/after: with enabled_tenants: "", no tenant was ever
compacted by any compactor replica (every tenant matched
"not owned by this shard"/was filtered out before ownership was even
checked). After this fix, an explicitly empty enabled_tenants (or
disabled_tenants) behaves identically to the field being omitted:
all tenants are eligible, subject to normal ring sharding.

Fixes #6581

Signed-off-by: Pujitha Paladugu 10557236+pujitha24@users.noreply.github.com

Copilot AI review requested due to automatic review settings July 22, 2026 02:29

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request fixes a configuration edge case where CSV-backed string-slice flags/YAML fields (notably compactor.enabled_tenants / -compactor.enabled-tenants) treated an explicitly empty string ("") as a one-element slice containing "", which inadvertently enabled allow-list mode for a nonexistent empty tenant and caused all real tenants to be skipped.

Changes:

  • Update StringSliceCSV.Set to special-case s == "" by resetting the slice to nil instead of using strings.Split, aligning behavior with “unset/omitted” semantics.
  • Add unit tests covering both direct Set("") behavior and YAML unmarshalling of csv: "".
  • Add a CHANGELOG bugfix entry documenting the user-facing impact and resolution.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
pkg/util/flagext/stringslicecsv.go Treat empty input string as an empty/unset list by setting the slice to nil before returning.
pkg/util/flagext/stringslicecsv_test.go Add tests to ensure empty-string inputs don’t produce [""] and YAML "" unmarshals to an empty list.
CHANGELOG.md Document the bugfix and its impact on enabled/disabled tenant CSV lists.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@pujitha24

Copy link
Copy Markdown
Contributor Author

The failing checks here are not caused by this diff — flagging in case that is what is holding it up.

Both failures are amd64-only (arm64 passes) and in code this PR does not touch: TestPartitionCompactor_ShouldCompactOnlyUsersOwnedByTheInstanceOnShardingEnabledAndMultipleInstancesRunning, a ring-sharding timing test in pkg/compactor.

Happy to rebase to re-trigger CI if that would help, or to dig further if you think I have this wrong.

Comment thread CHANGELOG.md Outdated
* [BUGFIX] Memberlist: Add `-memberlist.packet-read-timeout`, `-memberlist.max-packet-size`, and `-memberlist.max-concurrent-connections` flags to bound inbound gossip TCP connections, preventing slow-read, OOM, and connection-flood attacks on the gossip port. #7518
* [BUGFIX] Distributor: Add `WrappedHistogram` with configurable size limit (`-validation.max-native-histogram-size-bytes`, default 16 KB) to cap native histogram protobuf size before unmarshalling, preventing memory amplification attacks via packed varint deltas. #7570
* [BUGFIX] Distributor: Fix a panic (`slice bounds out of range`) in the stream push path when the context deadline expires while the worker goroutine is still marshalling a `WriteRequest`. #7541
* [BUGFIX] Config: Fix CSV-list flags/YAML fields (e.g. `-compactor.enabled-tenants`, `-compactor.disabled-tenants`) treating an explicitly empty string (`""`) as a one-element list containing an empty tenant name instead of an empty list. This caused every tenant to be skipped when `enabled_tenants: ""` was set in config, since only the (nonexistent) empty-string tenant matched the allow-list. #6581

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing it.

Two nits before merging:

  • Reference the PR number (#7714) rather than the issue.
  • The entry could also be trimmed to a sentence or two.

@pujitha24
pujitha24 requested a review from a team as a code owner September 7, 2026 01:33
@pujitha24
pujitha24 requested a review from alanprot September 7, 2026 01:33

@SungJin1212 SungJin1212 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@pujitha24

Copy link
Copy Markdown
Contributor Author

This is approved with green CI and no outstanding comments as far as I can tell — ready whenever you have a moment. Happy to rebase first if you'd like it freshened.

Comment thread CHANGELOG.md Outdated
* [BUGFIX] Memberlist: Add `-memberlist.packet-read-timeout`, `-memberlist.max-packet-size`, and `-memberlist.max-concurrent-connections` flags to bound inbound gossip TCP connections, preventing slow-read, OOM, and connection-flood attacks on the gossip port. #7518
* [BUGFIX] Distributor: Add `WrappedHistogram` with configurable size limit (`-validation.max-native-histogram-size-bytes`, default 16 KB) to cap native histogram protobuf size before unmarshalling, preventing memory amplification attacks via packed varint deltas. #7570
* [BUGFIX] Distributor: Fix a panic (`slice bounds out of range`) in the stream push path when the context deadline expires while the worker goroutine is still marshalling a `WriteRequest`. #7541
* [BUGFIX] Config: Fix CSV-list flags/YAML fields (e.g. `-compactor.enabled-tenants`) treating an explicitly empty string as a one-element list containing an empty tenant name instead of an empty list. #7714

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you move it to the ## master / unreleased section?

@pujitha24

Copy link
Copy Markdown
Contributor Author

Moved it to master / unreleased.

@SungJin1212

Copy link
Copy Markdown
Member

@pujitha24
Can you get a rebase?

Motivation:
Issue cortexproject#6581 reports that setting `compactor.enabled_tenants: ""`
explicitly in config causes the compactor to skip every tenant,
instead of behaving like the unset/omitted case (all tenants
allowed). The compactor logs "skipping user because it is not owned
by this shard" for every real tenant.

Approach:
`StringSliceCSV.Set` (pkg/util/flagext/stringslicecsv.go), which
backs `-compactor.enabled-tenants`/`-compactor.disabled-tenants` and
several other CSV-list flags (alertmanager, ruler, store-gateway,
ingester, querier, distributor, ring, limits, etc.), implemented
`Set` as `strings.Split(s, ",")`. Go's `strings.Split("", ",")`
returns `[""]`, a one-element slice containing an empty string, not
an empty/nil slice. YAML unmarshaling for `enabled_tenants: ""`
calls `Set("")`, so `EnabledTenants` ends up as `[""]` with
`len() == 1`.

`users.NewAllowedTenants` (pkg/util/users/allowed_tenants.go) treats
`len(enabled) > 0` as "only tenants in this list are allowed", so
with `enabled == [""]` only a tenant literally named `""` is
allowed and every real tenant is rejected. This same
`NewAllowedTenants` helper backs the identical enabled/disabled
tenant list pattern in the compactor, alertmanager, ruler, and
store-gateway, so all four are affected identically by explicitly
empty tenant-list config, though issue cortexproject#6581 only reports it for the
compactor.

Fix `Set` to special-case an empty input string by setting the value
to nil, matching the sibling `SecretStringSliceCSV.Set`, which
already special-cases `s == ""` this same way.

Validation:
  go build ./...
  go test ./pkg/util/flagext/... ./pkg/compactor/... ./pkg/util/validation/... ./pkg/util/users/... -tags "netgo slicelabels"
All pass, including new tests in pkg/util/flagext/stringslicecsv_test.go
covering direct `Set("")` and YAML unmarshaling of `csv: ""`.

Before/after: with `enabled_tenants: ""`, no tenant was ever
compacted by any compactor replica (every tenant matched
"not owned by this shard"/was filtered out before ownership was even
checked). After this fix, an explicitly empty `enabled_tenants` (or
`disabled_tenants`) behaves identically to the field being omitted:
all tenants are eligible, subject to normal ring sharding.

Fixes cortexproject#6581

Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
The entry landed under the already-released 1.21.1 section instead
of master / unreleased.

Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
@SungJin1212
SungJin1212 merged commit 71e7338 into cortexproject:master Sep 8, 2026
40 checks passed
CharlieTLe added a commit to CharlieTLe/cortex that referenced this pull request Sep 11, 2026
Get the unreleased section into the shape RELEASE.md asks for before the
release-1.22 cut, so operators reading the notes see the changes that affect them:

- Delete the stale duplicate of cortexproject#7375, which already shipped in 1.21.0 as cortexproject#7370.
- Re-sort into CHANGE -> FEATURE -> ENHANCEMENT -> BUGFIX.
- Correct the gRPC entry: the bump landed at v1.82.1, not v1.79.3.
- Fold follow-up PRs into the entry they belong to.
- Add three missing user-facing entries: cortexproject#7513, cortexproject#7514 and cortexproject#7559.
- Reclassify as CHANGE the entries that break existing configs or log consumers:
  the sign-key validation (cortexproject#7587), the Alertmanager per-tenant *_file rejections
  (cortexproject#7767, cortexproject#7768, now one entry) and the time_taken -> time_taken_ms rename (cortexproject#7649).
- Note the operator impact of the distroless base image (cortexproject#7637) and of the
  500 -> 499 reclassification (cortexproject#7717).

Rebased onto master, which added eight entries after this was first written.
They are curated the same way:

- Sorted into their categories: the deprecated flag removal (cortexproject#7790) and the
  max-exemplars deprecation (cortexproject#7793) under CHANGE, the X-Grafana-User query log
  (cortexproject#7799) under ENHANCEMENT, the YAML zero-value validation (cortexproject#7700) and the
  ReadPartitionedGroupInfo error handling (cortexproject#7766) under BUGFIX.
- Folded the Go toolchain bump (cortexproject#7807, cortexproject#7814) into the existing build image
  entry, which now reads 1.27.0 rather than carrying a second entry for it.
- Folded cortexproject#7745 into cortexproject#7698: both are the same wipe-on-transient-DNS-failure bug,
  cortexproject#7698 on the A record path and cortexproject#7745 on the SRV path.
- Folded cortexproject#7743 into cortexproject#7640: both are panics in the active request tracker's
  truncation of match[]/query values.

Rebased again onto master, which added eight more entries. Same treatment:

- Sorted into their categories: the evaluation-delay-duration removal (cortexproject#7792)
  and the fifocache/ingester-metadata-streaming removals (cortexproject#7791) under CHANGE,
  the parquet max-block-label-names limit (cortexproject#7625), the non-pointer
  HistogramBucket slice (cortexproject#7809) and the merge iterator BatchSize (cortexproject#7823) under
  ENHANCEMENT, and the CSV-list empty-string fix (cortexproject#7714) under BUGFIX.
- Folded the Thanos/promql-engine refresh (cortexproject#7788) into the existing upgrade
  entry, which already carries cortexproject#7691, cortexproject#7505 and cortexproject#7740.

Signed-off-by: Charlie Le <charlie_le@apple.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tenants are getting skipp

3 participants