feat(fluentbit): Support network relevant settings for s3 output#1955
Conversation
There was a problem hiding this comment.
Pull request overview
Adds Fluent Bit networking/connection-management options (timeouts, DNS preferences, keepalive, TCP keepalive, source address, proxy env handling, worker connections) to the S3 output plugin, exposing them via the CRDs and Helm charts and adding test coverage.
Changes:
- Add ~15 new networking fields to the
S3struct and emit them asnet.*params inS3.Params. - Regenerate the affected CRD YAMLs, Helm chart templates, deepcopy, and API docs.
- Extend
TestOutput_S3_Paramsto cover the new fields.
Reviewed changes
Copilot reviewed 11 out of 12 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| apis/fluentbit/v1alpha2/plugins/output/s3_types.go | Adds new networking fields and maps them to net.* params |
| apis/fluentbit/v1alpha2/plugins/output/s3_types_test.go | Tests new fields produce the expected net.* KVs |
| apis/fluentbit/v1alpha2/plugins/output/zz_generated.deepcopy.go | Generated deepcopy for new S3 pointer fields |
| config/crd/bases/fluentbit.fluent.io_outputs.yaml | Regenerated CRD with new S3 fields |
| config/crd/bases/fluentbit.fluent.io_clusteroutputs.yaml | Regenerated CRD with new S3 fields |
| charts/fluent-operator/crds/fluentbit.fluent.io_outputs.yaml | Helm CRD chart updated |
| charts/fluent-operator/crds/fluentbit.fluent.io_clusteroutputs.yaml | Helm CRD chart updated |
| charts/fluent-operator-fluent-bit-crds/templates/fluentbit.fluent.io_outputs.yaml | Helm CRD chart updated |
| charts/fluent-operator-fluent-bit-crds/templates/fluentbit.fluent.io_clusteroutputs.yaml | Helm CRD chart updated |
| manifests/setup/setup.yaml | Generated manifest updated |
| manifests/setup/fluent-operator-crd.yaml | Generated manifest updated |
| docs/plugins/fluentbit/output/s3.md | Doc lists new fields |
Files not reviewed (1)
- apis/fluentbit/v1alpha2/plugins/output/zz_generated.deepcopy.go: Language not supported
Comments suppressed due to low confidence (1)
apis/fluentbit/v1alpha2/plugins/output/s3_types.go:150
Keepaliveis documented as a boolean on/off switch for connection keepalive support, but it is being emitted asnet.tcp_keepalive. Per Fluent Bit's networking documentation (and the existingplugins.Networking.Paramsimplementation atapis/fluentbit/v1alpha2/plugins/net_types.go:55-56), the on/off keepalive toggle is thenet.keepalivesetting.net.tcp_keepaliveis a different (TCP-level) option. As written, enablingkeepalive: "on"on S3 will set the wrong Fluent Bit parameter.
plugins.InsertKVField(kvs, "net.tcp_keepalive", o.Keepalive)
|
Hi @smallc2009 can you please help resolve the conflicts? Thank you. |
cce8ea9 to
ac49398
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 10 out of 11 changed files in this pull request and generated 9 comments.
Files not reviewed (1)
- apis/fluentbit/v1alpha2/plugins/output/zz_generated.deepcopy.go: Language not supported
Comments suppressed due to low confidence (2)
manifests/setup/setup.yaml:1
- Fix grammar/casing in descriptions: change "this include" to "this includes", and "IPV6" to "IPv6" for consistency and correctness in user-facing CRD docs.
manifests/setup/setup.yaml:1 - Fix grammar/casing in descriptions: change "this include" to "this includes", and "IPV6" to "IPv6" for consistency and correctness in user-facing CRD docs.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 11 out of 14 changed files in this pull request and generated 2 comments.
Files not reviewed (2)
- apis/fluentbit/v1alpha2/plugins/output/zz_generated.deepcopy.go: Generated file
- apis/fluentbit/v1alpha2/plugins/zz_generated.deepcopy.go: Generated file
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 11 out of 14 changed files in this pull request and generated 4 comments.
Files not reviewed (2)
- apis/fluentbit/v1alpha2/plugins/output/zz_generated.deepcopy.go: Generated file
- apis/fluentbit/v1alpha2/plugins/zz_generated.deepcopy.go: Generated file
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 11 out of 14 changed files in this pull request and generated 3 comments.
Files not reviewed (2)
- apis/fluentbit/v1alpha2/plugins/output/zz_generated.deepcopy.go: Generated file
- apis/fluentbit/v1alpha2/plugins/zz_generated.deepcopy.go: Generated file
Signed-off-by: Anson <anson.liu@live.com>
0690e75 to
c716ea0
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 16 changed files in this pull request and generated 3 comments.
Files not reviewed (2)
- apis/fluentbit/v1alpha2/plugins/output/zz_generated.deepcopy.go: Generated file
- apis/fluentbit/v1alpha2/plugins/zz_generated.deepcopy.go: Generated file
Signed-off-by: Anson <anson.liu@live.com>
Signed-off-by: Anson <anson.liu@live.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 26 out of 30 changed files in this pull request and generated 2 comments.
Files not reviewed (2)
- apis/fluentbit/v1alpha2/plugins/output/zz_generated.deepcopy.go: Generated file
- apis/fluentbit/v1alpha2/plugins/zz_generated.deepcopy.go: Generated file
…tion Signed-off-by: Anson <anson.liu@live.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 26 out of 30 changed files in this pull request and generated 1 comment.
Files not reviewed (2)
- apis/fluentbit/v1alpha2/plugins/output/zz_generated.deepcopy.go: Generated file
- apis/fluentbit/v1alpha2/plugins/zz_generated.deepcopy.go: Generated file
What this PR does / why we need it:
This pull request extends the Fluent Bit S3 output plugin and its networking configuration with several new networking options, improves maintainability by refactoring parameter handling, and updates documentation and tests accordingly. The main focus is on enhancing the flexibility and control over network-related settings for the S3 output.
Networking enhancements:
Networkingstruct:IoTimeout,DNSPreferIPv6,ProxyEnvIgnore,TCPKeepalive,TCPKeepaliveTime,TCPKeepaliveInterval, andTCPKeepaliveProbes, allowing more granular control over network timeouts, DNS preferences, proxy handling, and TCP keepalive behavior. [1] [2]S3struct) to support an optionalNetworkingfield, enabling users to specify advanced networking options for S3 output.Codebase improvements:
Networking.Paramsmethod to use theInsertKVFieldhelper for more concise and maintainable parameter insertion, replacing repetitive manual code.NetworkingandS3structs to ensure all new fields are correctly copied. [1] [2] [3] [4]Documentation and testing:
Networkingstruct and S3 output plugin to describe the new networking fields and their usage. [1] [2]Which issue(s) this PR fixes:
Fixes #1930
Does this PR introduced a user-facing change?
Additional documentation, usage docs, etc.:
an sample configuration