Fix StringSliceCSV treating explicit empty string as a one-element list - #7714
Conversation
There was a problem hiding this comment.
Pull request overview
This pull request fixes a configuration edge case where CSV-backed string-slice flags/YAML fields (notably compactor.enabled_tenants / -compactor.enabled-tenants) treated an explicitly empty string ("") as a one-element slice containing "", which inadvertently enabled allow-list mode for a nonexistent empty tenant and caused all real tenants to be skipped.
Changes:
- Update
StringSliceCSV.Setto special-cases == ""by resetting the slice tonilinstead of usingstrings.Split, aligning behavior with “unset/omitted” semantics. - Add unit tests covering both direct
Set("")behavior and YAML unmarshalling ofcsv: "". - Add a CHANGELOG bugfix entry documenting the user-facing impact and resolution.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| pkg/util/flagext/stringslicecsv.go | Treat empty input string as an empty/unset list by setting the slice to nil before returning. |
| pkg/util/flagext/stringslicecsv_test.go | Add tests to ensure empty-string inputs don’t produce [""] and YAML "" unmarshals to an empty list. |
| CHANGELOG.md | Document the bugfix and its impact on enabled/disabled tenant CSV lists. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
2ea8b1d to
1a8580a
Compare
|
The failing checks here are not caused by this diff — flagging in case that is what is holding it up. Both failures are amd64-only (arm64 passes) and in code this PR does not touch: Happy to rebase to re-trigger CI if that would help, or to dig further if you think I have this wrong. |
| * [BUGFIX] Memberlist: Add `-memberlist.packet-read-timeout`, `-memberlist.max-packet-size`, and `-memberlist.max-concurrent-connections` flags to bound inbound gossip TCP connections, preventing slow-read, OOM, and connection-flood attacks on the gossip port. #7518 | ||
| * [BUGFIX] Distributor: Add `WrappedHistogram` with configurable size limit (`-validation.max-native-histogram-size-bytes`, default 16 KB) to cap native histogram protobuf size before unmarshalling, preventing memory amplification attacks via packed varint deltas. #7570 | ||
| * [BUGFIX] Distributor: Fix a panic (`slice bounds out of range`) in the stream push path when the context deadline expires while the worker goroutine is still marshalling a `WriteRequest`. #7541 | ||
| * [BUGFIX] Config: Fix CSV-list flags/YAML fields (e.g. `-compactor.enabled-tenants`, `-compactor.disabled-tenants`) treating an explicitly empty string (`""`) as a one-element list containing an empty tenant name instead of an empty list. This caused every tenant to be skipped when `enabled_tenants: ""` was set in config, since only the (nonexistent) empty-string tenant matched the allow-list. #6581 |
There was a problem hiding this comment.
Thanks for fixing it.
Two nits before merging:
- Reference the PR number (
#7714) rather than the issue. - The entry could also be trimmed to a sentence or two.
|
This is approved with green CI and no outstanding comments as far as I can tell — ready whenever you have a moment. Happy to rebase first if you'd like it freshened. |
| * [BUGFIX] Memberlist: Add `-memberlist.packet-read-timeout`, `-memberlist.max-packet-size`, and `-memberlist.max-concurrent-connections` flags to bound inbound gossip TCP connections, preventing slow-read, OOM, and connection-flood attacks on the gossip port. #7518 | ||
| * [BUGFIX] Distributor: Add `WrappedHistogram` with configurable size limit (`-validation.max-native-histogram-size-bytes`, default 16 KB) to cap native histogram protobuf size before unmarshalling, preventing memory amplification attacks via packed varint deltas. #7570 | ||
| * [BUGFIX] Distributor: Fix a panic (`slice bounds out of range`) in the stream push path when the context deadline expires while the worker goroutine is still marshalling a `WriteRequest`. #7541 | ||
| * [BUGFIX] Config: Fix CSV-list flags/YAML fields (e.g. `-compactor.enabled-tenants`) treating an explicitly empty string as a one-element list containing an empty tenant name instead of an empty list. #7714 |
There was a problem hiding this comment.
Could you move it to the ## master / unreleased section?
|
Moved it to master / unreleased. |
|
@pujitha24 |
Motivation: Issue cortexproject#6581 reports that setting `compactor.enabled_tenants: ""` explicitly in config causes the compactor to skip every tenant, instead of behaving like the unset/omitted case (all tenants allowed). The compactor logs "skipping user because it is not owned by this shard" for every real tenant. Approach: `StringSliceCSV.Set` (pkg/util/flagext/stringslicecsv.go), which backs `-compactor.enabled-tenants`/`-compactor.disabled-tenants` and several other CSV-list flags (alertmanager, ruler, store-gateway, ingester, querier, distributor, ring, limits, etc.), implemented `Set` as `strings.Split(s, ",")`. Go's `strings.Split("", ",")` returns `[""]`, a one-element slice containing an empty string, not an empty/nil slice. YAML unmarshaling for `enabled_tenants: ""` calls `Set("")`, so `EnabledTenants` ends up as `[""]` with `len() == 1`. `users.NewAllowedTenants` (pkg/util/users/allowed_tenants.go) treats `len(enabled) > 0` as "only tenants in this list are allowed", so with `enabled == [""]` only a tenant literally named `""` is allowed and every real tenant is rejected. This same `NewAllowedTenants` helper backs the identical enabled/disabled tenant list pattern in the compactor, alertmanager, ruler, and store-gateway, so all four are affected identically by explicitly empty tenant-list config, though issue cortexproject#6581 only reports it for the compactor. Fix `Set` to special-case an empty input string by setting the value to nil, matching the sibling `SecretStringSliceCSV.Set`, which already special-cases `s == ""` this same way. Validation: go build ./... go test ./pkg/util/flagext/... ./pkg/compactor/... ./pkg/util/validation/... ./pkg/util/users/... -tags "netgo slicelabels" All pass, including new tests in pkg/util/flagext/stringslicecsv_test.go covering direct `Set("")` and YAML unmarshaling of `csv: ""`. Before/after: with `enabled_tenants: ""`, no tenant was ever compacted by any compactor replica (every tenant matched "not owned by this shard"/was filtered out before ownership was even checked). After this fix, an explicitly empty `enabled_tenants` (or `disabled_tenants`) behaves identically to the field being omitted: all tenants are eligible, subject to normal ring sharding. Fixes cortexproject#6581 Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
The entry landed under the already-released 1.21.1 section instead of master / unreleased. Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
c4d812f to
3a425d8
Compare
Get the unreleased section into the shape RELEASE.md asks for before the release-1.22 cut, so operators reading the notes see the changes that affect them: - Delete the stale duplicate of cortexproject#7375, which already shipped in 1.21.0 as cortexproject#7370. - Re-sort into CHANGE -> FEATURE -> ENHANCEMENT -> BUGFIX. - Correct the gRPC entry: the bump landed at v1.82.1, not v1.79.3. - Fold follow-up PRs into the entry they belong to. - Add three missing user-facing entries: cortexproject#7513, cortexproject#7514 and cortexproject#7559. - Reclassify as CHANGE the entries that break existing configs or log consumers: the sign-key validation (cortexproject#7587), the Alertmanager per-tenant *_file rejections (cortexproject#7767, cortexproject#7768, now one entry) and the time_taken -> time_taken_ms rename (cortexproject#7649). - Note the operator impact of the distroless base image (cortexproject#7637) and of the 500 -> 499 reclassification (cortexproject#7717). Rebased onto master, which added eight entries after this was first written. They are curated the same way: - Sorted into their categories: the deprecated flag removal (cortexproject#7790) and the max-exemplars deprecation (cortexproject#7793) under CHANGE, the X-Grafana-User query log (cortexproject#7799) under ENHANCEMENT, the YAML zero-value validation (cortexproject#7700) and the ReadPartitionedGroupInfo error handling (cortexproject#7766) under BUGFIX. - Folded the Go toolchain bump (cortexproject#7807, cortexproject#7814) into the existing build image entry, which now reads 1.27.0 rather than carrying a second entry for it. - Folded cortexproject#7745 into cortexproject#7698: both are the same wipe-on-transient-DNS-failure bug, cortexproject#7698 on the A record path and cortexproject#7745 on the SRV path. - Folded cortexproject#7743 into cortexproject#7640: both are panics in the active request tracker's truncation of match[]/query values. Rebased again onto master, which added eight more entries. Same treatment: - Sorted into their categories: the evaluation-delay-duration removal (cortexproject#7792) and the fifocache/ingester-metadata-streaming removals (cortexproject#7791) under CHANGE, the parquet max-block-label-names limit (cortexproject#7625), the non-pointer HistogramBucket slice (cortexproject#7809) and the merge iterator BatchSize (cortexproject#7823) under ENHANCEMENT, and the CSV-list empty-string fix (cortexproject#7714) under BUGFIX. - Folded the Thanos/promql-engine refresh (cortexproject#7788) into the existing upgrade entry, which already carries cortexproject#7691, cortexproject#7505 and cortexproject#7740. Signed-off-by: Charlie Le <charlie_le@apple.com>
Motivation:
Issue #6581 reports that setting
compactor.enabled_tenants: ""explicitly in config causes the compactor to skip every tenant,
instead of behaving like the unset/omitted case (all tenants
allowed). The compactor logs "skipping user because it is not owned
by this shard" for every real tenant.
Approach:
StringSliceCSV.Set(pkg/util/flagext/stringslicecsv.go), whichbacks
-compactor.enabled-tenants/-compactor.disabled-tenantsandseveral other CSV-list flags (alertmanager, ruler, store-gateway,
ingester, querier, distributor, ring, limits, etc.), implemented
Setasstrings.Split(s, ","). Go'sstrings.Split("", ",")returns
[""], a one-element slice containing an empty string, notan empty/nil slice. YAML unmarshaling for
enabled_tenants: ""calls
Set(""), soEnabledTenantsends up as[""]withlen() == 1.users.NewAllowedTenants(pkg/util/users/allowed_tenants.go) treatslen(enabled) > 0as "only tenants in this list are allowed", sowith
enabled == [""]only a tenant literally named""isallowed and every real tenant is rejected. This same
NewAllowedTenantshelper backs the identical enabled/disabledtenant list pattern in the compactor, alertmanager, ruler, and
store-gateway, so all four are affected identically by explicitly
empty tenant-list config, though issue #6581 only reports it for the
compactor.
Fix
Setto special-case an empty input string by setting the valueto nil, matching the sibling
SecretStringSliceCSV.Set, whichalready special-cases
s == ""this same way.Validation:
go build ./...
go test ./pkg/util/flagext/... ./pkg/compactor/... ./pkg/util/validation/... ./pkg/util/users/... -tags "netgo slicelabels"
All pass, including new tests in pkg/util/flagext/stringslicecsv_test.go
covering direct
Set("")and YAML unmarshaling ofcsv: "".Before/after: with
enabled_tenants: "", no tenant was evercompacted by any compactor replica (every tenant matched
"not owned by this shard"/was filtered out before ownership was even
checked). After this fix, an explicitly empty
enabled_tenants(ordisabled_tenants) behaves identically to the field being omitted:all tenants are eligible, subject to normal ring sharding.
Fixes #6581
Signed-off-by: Pujitha Paladugu 10557236+pujitha24@users.noreply.github.com