Skip to content

feat(fluentbit): Support network relevant settings for s3 output#1955

Merged
cw-Guo merged 5 commits into
fluent:masterfrom
smallc2009:feature/support-options-for-s3-output
Jun 16, 2026
Merged

feat(fluentbit): Support network relevant settings for s3 output#1955
cw-Guo merged 5 commits into
fluent:masterfrom
smallc2009:feature/support-options-for-s3-output

Conversation

@smallc2009

@smallc2009 smallc2009 commented May 18, 2026

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

This pull request extends the Fluent Bit S3 output plugin and its networking configuration with several new networking options, improves maintainability by refactoring parameter handling, and updates documentation and tests accordingly. The main focus is on enhancing the flexibility and control over network-related settings for the S3 output.

Networking enhancements:

  • Added new fields to the Networking struct: IoTimeout, DNSPreferIPv6, ProxyEnvIgnore, TCPKeepalive, TCPKeepaliveTime, TCPKeepaliveInterval, and TCPKeepaliveProbes, allowing more granular control over network timeouts, DNS preferences, proxy handling, and TCP keepalive behavior. [1] [2]
  • Updated the S3 output plugin (S3 struct) to support an optional Networking field, enabling users to specify advanced networking options for S3 output.

Codebase improvements:

  • Refactored the Networking.Params method to use the InsertKVField helper for more concise and maintainable parameter insertion, replacing repetitive manual code.
  • Updated deepcopy methods for both Networking and S3 structs to ensure all new fields are correctly copied. [1] [2] [3] [4]

Documentation and testing:

  • Updated documentation for the Networking struct and S3 output plugin to describe the new networking fields and their usage. [1] [2]
  • Enhanced the S3 output plugin test to cover the new networking options and verify correct parameter generation. [1] [2]
  • Ensured the S3 plugin merges networking parameters into its configuration if provided.

Which issue(s) this PR fixes:

Fixes #1930

Does this PR introduced a user-facing change?


Additional documentation, usage docs, etc.:

an sample configuration

apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterOutput
metadata:
  name: s3-output-sample
  labels:
    fluentbit.fluent.io/enabled: "true"
spec:
  match: kube.*
  s3:
    Region: us-east-1
    Bucket: fluent-operator-test-logs
    Endpoint: https://s3.amazonaws.com
    TotalFileSize: 5M
    UploadTimeout: 1m
    S3KeyFormat: /fluent-bit-logs/$TAG/%Y/%m/%d/%H/%M/%S/$UUID.gz
    Compression: gzip
    UsePutObject: true
    networking:
      DNSMode: UDP
      DNSResolver: LEGACY
      DNSPreferIPv4: true
      connectTimeout: 10
      ioTimeout: 30
      keepalive: "on"
      keepaliveIdleTimeout: 30
      maxWorkerConnections: 0

Copilot AI review requested due to automatic review settings May 18, 2026 07:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds Fluent Bit networking/connection-management options (timeouts, DNS preferences, keepalive, TCP keepalive, source address, proxy env handling, worker connections) to the S3 output plugin, exposing them via the CRDs and Helm charts and adding test coverage.

Changes:

  • Add ~15 new networking fields to the S3 struct and emit them as net.* params in S3.Params.
  • Regenerate the affected CRD YAMLs, Helm chart templates, deepcopy, and API docs.
  • Extend TestOutput_S3_Params to cover the new fields.

Reviewed changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
apis/fluentbit/v1alpha2/plugins/output/s3_types.go Adds new networking fields and maps them to net.* params
apis/fluentbit/v1alpha2/plugins/output/s3_types_test.go Tests new fields produce the expected net.* KVs
apis/fluentbit/v1alpha2/plugins/output/zz_generated.deepcopy.go Generated deepcopy for new S3 pointer fields
config/crd/bases/fluentbit.fluent.io_outputs.yaml Regenerated CRD with new S3 fields
config/crd/bases/fluentbit.fluent.io_clusteroutputs.yaml Regenerated CRD with new S3 fields
charts/fluent-operator/crds/fluentbit.fluent.io_outputs.yaml Helm CRD chart updated
charts/fluent-operator/crds/fluentbit.fluent.io_clusteroutputs.yaml Helm CRD chart updated
charts/fluent-operator-fluent-bit-crds/templates/fluentbit.fluent.io_outputs.yaml Helm CRD chart updated
charts/fluent-operator-fluent-bit-crds/templates/fluentbit.fluent.io_clusteroutputs.yaml Helm CRD chart updated
manifests/setup/setup.yaml Generated manifest updated
manifests/setup/fluent-operator-crd.yaml Generated manifest updated
docs/plugins/fluentbit/output/s3.md Doc lists new fields
Files not reviewed (1)
  • apis/fluentbit/v1alpha2/plugins/output/zz_generated.deepcopy.go: Language not supported
Comments suppressed due to low confidence (1)

apis/fluentbit/v1alpha2/plugins/output/s3_types.go:150

  • Keepalive is documented as a boolean on/off switch for connection keepalive support, but it is being emitted as net.tcp_keepalive. Per Fluent Bit's networking documentation (and the existing plugins.Networking.Params implementation at apis/fluentbit/v1alpha2/plugins/net_types.go:55-56), the on/off keepalive toggle is the net.keepalive setting. net.tcp_keepalive is a different (TCP-level) option. As written, enabling keepalive: "on" on S3 will set the wrong Fluent Bit parameter.
	plugins.InsertKVField(kvs, "net.tcp_keepalive", o.Keepalive)

Comment thread apis/fluentbit/v1alpha2/plugins/output/s3_types.go Outdated
@joshuabaird joshuabaird changed the title Support network relevant settings for s3 output feat(fluentbit): Support network relevant settings for s3 output May 21, 2026
@cw-Guo

cw-Guo commented May 29, 2026

Copy link
Copy Markdown
Collaborator

Hi @smallc2009 can you please help resolve the conflicts? Thank you.

Copilot AI review requested due to automatic review settings May 29, 2026 04:05
@smallc2009 smallc2009 force-pushed the feature/support-options-for-s3-output branch from cce8ea9 to ac49398 Compare May 29, 2026 04:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 10 out of 11 changed files in this pull request and generated 9 comments.

Files not reviewed (1)
  • apis/fluentbit/v1alpha2/plugins/output/zz_generated.deepcopy.go: Language not supported
Comments suppressed due to low confidence (2)

manifests/setup/setup.yaml:1

  • Fix grammar/casing in descriptions: change "this include" to "this includes", and "IPV6" to "IPv6" for consistency and correctness in user-facing CRD docs.
    manifests/setup/setup.yaml:1
  • Fix grammar/casing in descriptions: change "this include" to "this includes", and "IPV6" to "IPv6" for consistency and correctness in user-facing CRD docs.

Comment thread apis/fluentbit/v1alpha2/plugins/output/s3_types.go Outdated
Comment thread apis/fluentbit/v1alpha2/plugins/output/s3_types.go Outdated
Comment thread apis/fluentbit/v1alpha2/plugins/output/s3_types.go Outdated
Comment thread apis/fluentbit/v1alpha2/plugins/output/s3_types.go Outdated
Comment thread apis/fluentbit/v1alpha2/plugins/output/s3_types.go Outdated
Comment thread apis/fluentbit/v1alpha2/plugins/output/s3_types.go Outdated
Comment thread apis/fluentbit/v1alpha2/plugins/output/s3_types.go Outdated
Comment thread docs/plugins/fluentbit/output/s3.md Outdated
Comment thread docs/plugins/fluentbit/output/s3.md Outdated
Comment thread apis/fluentbit/v1alpha2/plugins/output/s3_types.go Outdated
Copilot AI review requested due to automatic review settings June 13, 2026 23:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 14 changed files in this pull request and generated 2 comments.

Files not reviewed (2)
  • apis/fluentbit/v1alpha2/plugins/output/zz_generated.deepcopy.go: Generated file
  • apis/fluentbit/v1alpha2/plugins/zz_generated.deepcopy.go: Generated file

Comment thread docs/plugins/fluentbit/output/s3.md Outdated
Comment thread apis/fluentbit/v1alpha2/plugins/net_types.go Outdated
@smallc2009 smallc2009 requested review from Copilot and cw-Guo June 13, 2026 23:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 14 changed files in this pull request and generated 4 comments.

Files not reviewed (2)
  • apis/fluentbit/v1alpha2/plugins/output/zz_generated.deepcopy.go: Generated file
  • apis/fluentbit/v1alpha2/plugins/zz_generated.deepcopy.go: Generated file

Comment thread apis/fluentbit/v1alpha2/plugins/net_types.go Outdated
Comment thread docs/plugins/fluentbit/output/s3.md Outdated
Comment thread docs/plugins/fluentbit/net.md
Comment thread docs/plugins/fluentbit/net.md Outdated
Copilot AI review requested due to automatic review settings June 16, 2026 00:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 14 changed files in this pull request and generated 3 comments.

Files not reviewed (2)
  • apis/fluentbit/v1alpha2/plugins/output/zz_generated.deepcopy.go: Generated file
  • apis/fluentbit/v1alpha2/plugins/zz_generated.deepcopy.go: Generated file

Comment thread apis/fluentbit/v1alpha2/plugins/output/s3_types_test.go
Comment thread docs/plugins/fluentbit/output/s3.md Outdated
Comment thread apis/fluentbit/v1alpha2/plugins/net_types.go Outdated
Signed-off-by: Anson <anson.liu@live.com>
Copilot AI review requested due to automatic review settings June 16, 2026 00:39
@smallc2009 smallc2009 force-pushed the feature/support-options-for-s3-output branch from 0690e75 to c716ea0 Compare June 16, 2026 00:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 12 out of 16 changed files in this pull request and generated 3 comments.

Files not reviewed (2)
  • apis/fluentbit/v1alpha2/plugins/output/zz_generated.deepcopy.go: Generated file
  • apis/fluentbit/v1alpha2/plugins/zz_generated.deepcopy.go: Generated file

Comment thread apis/fluentbit/v1alpha2/plugins/output/s3_types_test.go
Comment thread apis/fluentbit/v1alpha2/plugins/net_types.go Outdated
Comment thread docs/plugins/fluentbit/output/s3.md Outdated
Copilot AI review requested due to automatic review settings June 16, 2026 01:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 26 out of 30 changed files in this pull request and generated 2 comments.

Files not reviewed (2)
  • apis/fluentbit/v1alpha2/plugins/output/zz_generated.deepcopy.go: Generated file
  • apis/fluentbit/v1alpha2/plugins/zz_generated.deepcopy.go: Generated file

Comment thread docs/plugins/fluentbit/net.md
Comment thread apis/fluentbit/v1alpha2/plugins/net_types.go
…tion

Signed-off-by: Anson <anson.liu@live.com>
Copilot AI review requested due to automatic review settings June 16, 2026 06:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 26 out of 30 changed files in this pull request and generated 1 comment.

Files not reviewed (2)
  • apis/fluentbit/v1alpha2/plugins/output/zz_generated.deepcopy.go: Generated file
  • apis/fluentbit/v1alpha2/plugins/zz_generated.deepcopy.go: Generated file

Comment thread apis/fluentbit/v1alpha2/plugins/output/s3_types_test.go
@cw-Guo cw-Guo merged commit d424f6c into fluent:master Jun 16, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

S3 networking options

3 participants