You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Update log collector docs for the fluentd to fluent-bit migration (EV-6164)
Calico Enterprise removes the fluentd log collector in favour of Fluent Bit
(calico-fluent-bit in calico-system; tigera/operator#4910). Update the
next-version Calico Enterprise and Calico Cloud docs to match:
- Rewrite "Filter flow logs" / "Filter DNS logs": examples move from
fluentd <filter> syntax to Fluent Bit YAML filter lists, the ConfigMap
is renamed fluentd-filters -> fluent-bit-filters, and an upgrade note
explains that old fluentd-syntax filters are not translated (the
operator raises a tigera status warning naming the offending key).
- Metrics pages: Fluent Bit's built-in Prometheus endpoint (port 2020,
/api/v2/metrics/prometheus, fluentbit_* metrics); the fluentd
buffer-space alert becomes fluentbit_output_chunk_available_capacity_percent
plus a dropped-chunks alert on fluentbit_output_retries_failed_total.
- BYO Prometheus: the fluent-bit tab needs no client TLS (plain HTTP
behind the allow-calico-fluent-bit policy); ServiceMonitor sample is
fluent-bit-metrics-service-monitor.yaml.
- Sweep of remaining pages: tigera-fluentd namespace -> calico-system,
fluentd-node -> calico-fluent-bit (labels, secrets, packet capture
retrieval via calico-node, diags output, resource override examples now
use calicoFluentBitDaemonSet, architecture/overview descriptions,
cc-arch-diagram ports, operator checklist).
Not touched: versioned docs, releases.json (historical), and the generated
installation API reference (regenerates from the operator release).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: calico-cloud/get-started/cc-arch-diagram.mdx
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -33,7 +33,7 @@ The following diagram shows the major components in a managed cluster, followed
33
33
| calico-node | Bundles key components that are required for networking containers with $[prodname]:<br/><br />• Felix<br />• BIRD<br />• confd |• TCP 5473 to Typha<br />• TCP 9900 and 9081 from Prometheus API service |
34
34
| Container threat detection | A threat detection engine that analyzes observed file and process activity to detect known malicious and suspicious activity. Monitors the following types of suspicious activity within containers:<br/><br />• Access to sensitive system files and directories<br />• Defense evasion<br />• Discovery<br />• Execution<br />• Persistence<br />• Privilege escalation<br/><br />Includes these components:<br /><br />**Runtime Security Operator**<br />An operator to manage and reconcile container threat defense components.<br/><br />**Runtime Reporter Pods**<br />Pods running on each node in the cluster to perform the detection activity outlined above.They send activity reports to Elasticsearch for analysis by $[prodname]. | TCP to Kubernetes API server |
35
35
| Compliance | Generates compliance reports for the Kubernetes cluster. Reports are based on archived flow and audit logs for Calico Cloud resources, plus any audit logs you’ve configured for Kubernetes resources in the Kubernetes API server. Compliance reports provide the following high-level information:<br /><br />• Endpoints explicitly protected using ingress or egress policy<br />• Policies and services<br /> - Policies and services associated with endpoints<br /> - Policy audit logs <br />• Traffic<br /> - Allowed ingress/egress traffic to/from namespaces, and to/from the internet Compliance includes these components: <br /><br />**compliance-snapshotter** <br />Handles listing of required Kubernetes and $[prodname]configuration and pushes snapshots to Elasticsearch. Snapshots give you visibility into configuration changes, and how the cluster-wide configuration has evolved within a reporting interval.<br /><br />**compliance-reporter**<br />Handles report generation. Reads configuration history from Elasticsearch and determines time evolution of cluster-wide configuration, including relationships between policies, endpoints, services, and network sets. Data is then passed through a zero-trust aggregator to determine the “worst-case outliers” in the reporting interval.<br /><br />**compliance-controller**<br />Reads report configuration and manages creation, deletion, and monitoring of report generation jobs.<br /><br />**compliance-benchmarker**<br />A daemonset that runs checks in the CIS Kubernetes Benchmark on each node so you can see if Kubernetes is securely deployed.<br /> | • TCP 8080 to Guardian<br />• TCP 6443 to Kubernetes API server |
36
-
|Fluentd | Open-source data collector for unified logging. Collects and forwards $[prodname] logs (flows, DNS, L7) to log storage. |• TCP 8080 to Guardian<br />• TCP 9080 from Prometheus API service |
36
+
|Fluent Bit| Open-source, lightweight log processor and forwarder. Collects and forwards $[prodname] logs (flows, DNS, L7) to log storage. |• TCP 8080 to Guardian<br />• TCP 2020 from Prometheus API service |
37
37
| Guardian | An agent running in each managed cluster that proxies communication between the $[prodname] tunnel server and your managed cluster. Secured using TLS tunnels.<br /> |• Port 9000 to tunnel server<br />• TCP 6443 to Kubernetes API server<br />• TCP 6443 from $[prodname] components |
38
38
| Installation endpoints | Endpoints at `*.calicocloud.io` and `*.projectcalico.org`. | TCP 443 for both |
39
39
| Intrusion detection controller | Handles integrations with threat intelligence feeds and $[prodname] custom alerts. |• TCP 8080 to Guardian<br />• TCP 6443 to Kubernetes API server |
@@ -42,7 +42,7 @@ The following diagram shows the major components in a managed cluster, followed
42
42
| kube-controllers | Monitors the Kubernetes API and performs actions based on cluster state. $[prodname] kube-controllers container includes these controllers:<br/><br />• Node<br />• Service<br />• Federated services<br />• Authorization<br /> |• TCP 9094 from Prometheus API service<br />• TCP 6443 to Kubernetes API server |
43
43
| Log storage | Storage for logs (flows, L7, DNS, audit). Data for each managed cluster is isolated and protected against unauthorized access. | n/a |
44
44
| Packet capture API | Retrieves capture files (pcap format) generated by a packet capture for use with network protocol analysis tools like Wireshark. Packet capture data is visible in the web console and Service Graph. |• TCP 8449 Guardian to Packet Capture API<br />• TCP 6443 to Kubernetes API server |
45
-
| Prometheus API service | Collects metrics from $[prodname] components and makes the metrics available to the web console. |• TCP 6443 to Kubernetes API server<br />• TCP 9080 to Fluentd<br />• TCP 9900 and 9081 to Prometheus API service |
45
+
| Prometheus API service | Collects metrics from $[prodname] components and makes the metrics available to the web console. |• TCP 6443 to Kubernetes API server<br />• TCP 2020 to Fluent Bit<br />• TCP 9900 and 9081 to Prometheus API service |
46
46
| Tigera API server | Allows users to manage $[prodname] resources such as policies and tiers through kubectl or the Kubernetes API server. |• TCP 9095 to Prometheus API service<br />• TCP 8080 from Kubernetes API server |
47
47
| Typha | Increases scale by reducing each node’s impact on the datastore. | TCP 5473 from calico-node to Typha |
48
48
| User access to the web console | Authenticated users can access the browser-based the web console, which provides network traffic visibility and troubleshooting, centralized multi-cluster management, threat-defense, container threat detection, policy lifecycle management, scan images for vulnerabilities, and compliance for multiple roles/stakeholders. | Port 443 to $[prodname] tunnel server |
`log-collector` collects flow and other logs and forwards them to $[prodname]. Check the pods and logs in the `tigera-fluentd` namespace. You should have one pod running on each of your nodes.
496
+
`log-collector` collects flow and other logs and forwards them to $[prodname]. Check the `calico-fluent-bit` pods and logs in the `calico-system` namespace. You should have one pod running on each of your nodes.
497
497
498
498
```bash
499
-
kubectl get pods -n tigera-fluentd
499
+
kubectl get pods -n calico-system -l k8s-app=calico-fluent-bit
500
500
```
501
501
502
502
```
503
-
NAME READY STATUS RESTARTS AGE
504
-
fluentd-node-5mzh6 1/1 Running 0 70m
505
-
fluentd-node-7vmxw 1/1 Running 0 70m
506
-
fluentd-node-bbc4p 1/1 Running 0 70m
507
-
fluentd-node-chfz4 1/1 Running 0 70m
508
-
fluentd-node-d6f56 1/1 Running 0 70m
503
+
NAME READY STATUS RESTARTS AGE
504
+
calico-fluent-bit-5mzh6 1/1 Running 0 70m
505
+
calico-fluent-bit-7vmxw 1/1 Running 0 70m
506
+
calico-fluent-bit-bbc4p 1/1 Running 0 70m
507
+
calico-fluent-bit-chfz4 1/1 Running 0 70m
508
+
calico-fluent-bit-d6f56 1/1 Running 0 70m
509
509
```
510
510
511
511
**7 - management-cluster-connection**
@@ -643,7 +643,7 @@ If cluster does not have enough capacity, it will not be able to deploy pods. Th
643
643
644
644
The high-level components $[prodname] needs to run are:
645
645
646
-
- Per node: 1 fluentd, 1 compliance benchmarker
646
+
- Per node: 1 fluent-bit, 1 compliance benchmarker
647
647
- On top of per node: 3 alertmanager (from statefulset), 1 prometheus, 1 prometheus operator, 1 kube-controllers, 2 compliance snapshotter and controller, 1 guardian, 1 ids controller, 1 apiserver
648
648
649
649
Some clusters have limited pod-networked pod capacity.
description: Suppress low-value Calico Cloud DNS log entries with Fluentd filters configured through a ConfigMap in the operator namespace of connected clusters.
2
+
description: Suppress low-value Calico Cloud DNS log entries with Fluent Bit filters configured through a ConfigMap in the operator namespace of connected clusters.
3
3
---
4
4
5
5
# Filter DNS logs
@@ -18,46 +18,53 @@ To enable DNS log filtering, follow these steps:
18
18
your desired filter using [Filter configuration files](#filter-configuration-files).
19
19
If you are also adding [flow filters](../flow/filtering.mdx) also add the `flow` file
20
20
to the directory.
21
-
1. Create the `fluentd-filters` ConfigMap in the `tigera-operator` namespace
21
+
1. Create the `fluent-bit-filters` ConfigMap in the `tigera-operator` namespace
Create your [fluentd filters](https://docs.fluentd.org/filter/grep).
56
+
Filters are written as a YAML list of [Fluent Bit filter](https://docs.fluentbit.io/manual/data-pipeline/filters/grep) entries. The `calico-fluent-bit` log collector ships the `grep`, `record_modifier`, `parser`, and `lua` filters. Filters you add under the `flow` key are applied to flow logs automatically; you do not need to set a `match` on each entry.
57
57
58
58
**Example: filter out a specific namespace**
59
59
60
-
This example filters out all flow logs whose source or destination namespace is "dev". Additional namespaces could be filtered by adjusting the regular expression "pattern"s, or by adding additional`exclude`blocks.
60
+
This example filters out all flow logs whose source or destination namespace is "dev". A record is dropped when it matches any of the `exclude` rules; additional namespaces could be filtered by adjusting the regular expressions, or by adding more`exclude`rules.
61
61
62
-
```
63
-
<filter flows>
64
-
@type grep
65
-
<exclude>
66
-
key source_namespace
67
-
pattern dev
68
-
</exclude>
69
-
<exclude>
70
-
key dest_namespace
71
-
pattern dev
72
-
</exclude>
73
-
</filter>
62
+
```yaml
63
+
- name: grep
64
+
exclude:
65
+
- source_namespace dev
66
+
- dest_namespace dev
74
67
```
75
68
76
69
**Example: filter out internet traffic to a specific deployment**
77
70
78
-
This example filters inbound internet traffic to the deployment with pods named, `nginx-internet-*`. Note the use of the `and`directive to filter out traffic that is both to the deployment, and from the internet (source `pub`).
71
+
This example filters inbound internet traffic to the deployment with pods named, `nginx-internet-*`. Note the use of `logical_op: and` to filter out only the traffic that is both to the deployment, and from the internet (source `pub`).
79
72
80
-
```
81
-
<filter flows>
82
-
@type grep
83
-
<and>
84
-
<exclude>
85
-
key dest_name_aggr
86
-
pattern ^nginx-internet
87
-
</exclude>
88
-
<exclude>
89
-
key source_name_aggr
90
-
pattern pub
91
-
</exclude>
92
-
</and>
93
-
</filter>
73
+
```yaml
74
+
- name: grep
75
+
logical_op: and
76
+
exclude:
77
+
- dest_name_aggr ^nginx-internet
78
+
- source_name_aggr pub
94
79
```
95
80
96
81
### Add filters to ConfigMap file
97
82
98
83
1. Create a `filters` directory with a file called `flow` with your desired filters. If you are also adding [dns filters](../dns/filtering-dns.mdx), add the `dns` file to the directory.
99
84
100
-
1. Create the `fluentd-filters` ConfigMap in the `tigera-operator` namespace with the following command.
85
+
1. Create the `fluent-bit-filters` ConfigMap in the `tigera-operator` namespace with the following command.
The operator inlines the filters into the log collector configuration and rolls the `calico-fluent-bit` daemonset automatically.
92
+
93
+
:::note Upgrading from a release that used fluentd
94
+
95
+
Earlier releases collected logs with fluentd and read filters in fluentd `<filter>` syntax from a ConfigMap named `fluentd-filters`. That ConfigMap is no longer read, and fluentd filter syntax cannot be translated automatically. Recreate your filters as Fluent Bit YAML filter lists under the new `fluent-bit-filters` name. If a filter key does not parse as Fluent Bit YAML, the operator skips that filter, reports a warning on the `tigera status` output naming the offending key, and continues to ship unfiltered logs.
Copy file name to clipboardExpand all lines: calico-cloud/observability/elastic/overview.mdx
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -45,7 +45,7 @@ Because of their high-volume, flow and dns logs support aggregation.
45
45
46
46
### Default log configuration and security
47
47
48
-
$[prodname] automatically installs fluentd on all nodes and collects flow, audit, and DNS logs. You can configure additional destinations like Amazon S3, Syslog, Splunk.
48
+
$[prodname] automatically installs Fluent Bit on all nodes and collects flow, audit, and DNS logs. You can configure additional destinations like Amazon S3, Syslog, Splunk.
49
49
50
50
$[prodname] enables user authentication in Elasticsearch, and secures access to Elasticsearch and Kibana instances using network policy.
0 commit comments