feat(cli): Add kubectl retina bpftrace command for real-time network issue tracing - #2061
Merged
Conversation
Alex Castilio (alexcastilio)
requested review from
Iti Agrawal (agrawaliti) and
Tahmid Alam (alam-tahmid)
February 13, 2026 19:11
Retina Code Coverage ReportTotal coverage increased from
|
| Impacted Files | Coverage | |
|---|---|---|
| cli/cmd/config.go | 0.0% ... 100.0% (100.0%) |
⬆️ |
| cli/cmd/trace.go | 0.0% ... 100.0% (100.0%) |
⬆️ |
| cli/cmd/version.go | 0.0% ... 100.0% (100.0%) |
⬆️ |
| cli/cmd/shell.go | 0.0% ... 65.2% (65.2%) |
⬆️ |
| cli/cmd/root.go | 0.0% ... 100.0% (100.0%) |
⬆️ |
kubectl retina bpftrace command for real-time network issue tracing
Alex Castilio (alexcastilio)
force-pushed
the
feat/drop-trace
branch
from
February 16, 2026 16:10
9c4d25f to
5cbcd00
Compare
Alex Castilio (alexcastilio)
force-pushed
the
feat/drop-trace
branch
from
February 18, 2026 12:05
b8779b0 to
db95cb9
Compare
Simone Rodigari (SRodi)
left a comment
Member
There was a problem hiding this comment.
The current kfree_skb logic parses every single dropped packet (including invalid ones, or ones valid but not matching filter) to extracting IPs before filtering, which is expensive. Using predicate filtering would be more efficient.
Alex Castilio (alexcastilio)
force-pushed
the
feat/drop-trace
branch
from
February 18, 2026 17:22
85486e0 to
edaf43f
Compare
…dation Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
…anagement Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
- ScriptGenerator creates bpftrace script for kfree_skb tracepoint - DropReasonsCommand() fetches drop reason enum from kernel at runtime - Outputs numeric reason codes (kernel-version specific) - Hex conversion for IP/CIDR filters prevents injection - Supports both table and JSON output formats - Use Args (not Command) to preserve entrypoint.sh debugfs mounting - Add required capabilities: MKNOD, SYS_CHROOT (per shell.md) - Add AppArmor unconfined profile for bpftrace - Handle duration timeout gracefully Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
…el retransmit reason code Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
…obe-specific semantics Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
…nt selection (--drops, --rst, --errors, --retransmits) Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
…ive in test result Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
…ssage Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
…UEUE drops with no consumer Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
…ignal.Stop cleanup Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Alex Castilio (alexcastilio)
force-pushed
the
feat/drop-trace
branch
from
February 25, 2026 14:55
e58c668 to
b409918
Compare
Copilot started reviewing on behalf of
Alex Castilio (alexcastilio)
February 25, 2026 15:05
View session
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an experimental kubectl retina bpftrace command to run bpftrace-based network issue tracing on a target Kubernetes node (via a host-network debug pod), with optional IP/CIDR filtering and table/JSON output.
Changes:
- Introduces
bpftraceCLI subcommand with flags for event selection, filtering, output format, and duration/startup timeout. - Implements bpftrace script generation (drops/RST/errors/retransmits/NFQUEUE) plus typed validation and unit tests.
- Adds troubleshooting documentation and E2E shell scripts for basic coverage and IP-filter behavior.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 14 comments.
Show a summary per file
| File | Description |
|---|---|
cli/cmd/bpftrace.go |
New bpftrace Cobra command, flag parsing, validation, and trace invocation. |
cli/cmd/bpftrace_test.go |
Unit tests for IP/CIDR/output validation and injection-pattern rejection. |
shell/trace.go |
Trace execution on a node via a host-network pod, capability setup, and exec streaming. |
shell/trace_test.go |
Unit tests for capabilities, TraceConfig typing, pod security context, and exec patterns. |
shell/tracescript.go |
bpftrace script generator, including injection-safe IP/CIDR filtering logic. |
shell/tracescript_test.go |
Unit tests for script contents, JSON/table output, and filter generation correctness. |
docs/06-Troubleshooting/bpftrace.md |
User-facing docs for the new command, event types, flags, and output examples. |
test/e2e/test_bpftrace_drops.sh |
E2E script to generate representative events and validate capture presence. |
test/e2e/test_bpftrace_filter_ip.sh |
E2E script validating --ip filtering correctness across event types. |
cli/README.md |
Mentions the new bpftrace subcommand in CLI README. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Alex Castilio (alexcastilio)
enabled auto-merge
February 25, 2026 15:44
Laksh (lakshk98)
pushed a commit
to lakshk98/retina
that referenced
this pull request
Aug 4, 2026
…k issue tracing (microsoft#2061) # Description ## Summary New `bpftrace` subcommand for real-time tracing of network issues on Kubernetes nodes using eBPF/bpftrace. ## Use Cases - Debug packet drops on a node (e.g., NetworkPolicy blocks, routing issues) - Trace TCP connection failures (RST sent/received, connection refused) - Identify retransmissions indicating packet loss or congestion - Filter events by IP or subnet to focus on specific endpoints ```bash # Trace all events (default) kubectl retina bpftrace <node> # Trace only drops and RSTs for a specific IP kubectl retina bpftrace <node> --drops --rst --ip 10.244.1.15 # Trace retransmits for a subnet kubectl retina bpftrace <node> --retransmits --cidr 10.244.0.0/16 ``` ## What's Implemented ### New CLI Command: `kubectl retina bpftrace <node-name>` Traces network issues on a specified Kubernetes node with the following capabilities: **Event Types Captured:** | Type | Probe | Description | |------|-------|-------------| | DROP | `kfree_skb` | Packet drops with kernel reason codes (e.g., NETFILTER_DROP for NetworkPolicy) | | RST_SENT | `tcp_send_reset` | TCP RST packets sent by this host | | RST_RECV | `tcp_receive_reset` | TCP RST packets received by this host | | SOCK_ERR | `inet_sk_error_report` | Socket errors (ECONNREFUSED, ETIMEDOUT, etc.) | | RETRANS | `tcp_retransmit_skb` | TCP retransmissions indicating packet loss | ## Flags | Flag | Description | |------|-------------| | `--ip` | Filter by IP address (src or dst) | | `--cidr` | Filter by CIDR (src or dst) | | `--drops` | Enable only packet drop events | | `--rst` | Enable only TCP RST events | | `--errors` | Enable only socket error events | | `--retransmits` | Enable only retransmit events | | `--all` | Enable all events (default) | | `--duration` | Trace duration (0 = until Ctrl-C) | | `--startup-timeout` | Pod startup timeout | | `-o, --output` | Output format: `table` or `json` | When no event flags are specified, all events are traced. ## Related Issue If this pull request is related to any issue, please mention it here. Additionally, make sure that the issue is assigned to you before submitting this pull request. ## Checklist - [ ] I have read the [contributing documentation](https://retina.sh/docs/Contributing/overview). - [ ] I signed and signed-off the commits (`git commit -S -s ...`). See [this documentation](https://docs.github.com/en/authentication/managing-commit-signature-verification/about-commit-signature-verification) on signing commits. - [ ] I have correctly attributed the author(s) of the code. - [ ] I have tested the changes locally. - [ ] I have followed the project's style guidelines. - [ ] I have updated the documentation, if necessary. - [ ] I have added tests, if applicable. ## Screenshots (if applicable) or Testing Completed <img width="956" height="893" alt="image" src="https://github.com/user-attachments/assets/cfefd21a-03a7-4518-88c5-66a28d3a4145" /> ## Additional Notes ## Limitations - **IPv4 only**: IPv6 not supported - **Linux only**: Windows nodes not supported - **Cilium CNI**: DROP events won't capture Cilium policy drops (Cilium uses eBPF datapath, not netfilter/kfree_skb) ## Testing ```bash # Build go build -o kubectl-retina ./cli # E2E test (validates all 4 event types) ./test/e2e/test_bpftrace_drops.sh # Manual NODE=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}') ./kubectl-retina bpftrace $NODE --duration 30s --retina-shell-image-version v1.0.3 ``` ## Security - IP/CIDR inputs validated and converted to hex (injection-safe) - Commands executed via array-based exec (no shell) - Pod uses minimal capabilities for bpftrace --- Please refer to the [CONTRIBUTING.md](../CONTRIBUTING.md) file for more information on how to contribute to this project. --------- Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Laksh (lakshk98)
pushed a commit
to lakshk98/retina
that referenced
this pull request
Aug 4, 2026
…k issue tracing (microsoft#2061) # Description ## Summary New `bpftrace` subcommand for real-time tracing of network issues on Kubernetes nodes using eBPF/bpftrace. ## Use Cases - Debug packet drops on a node (e.g., NetworkPolicy blocks, routing issues) - Trace TCP connection failures (RST sent/received, connection refused) - Identify retransmissions indicating packet loss or congestion - Filter events by IP or subnet to focus on specific endpoints ```bash # Trace all events (default) kubectl retina bpftrace <node> # Trace only drops and RSTs for a specific IP kubectl retina bpftrace <node> --drops --rst --ip 10.244.1.15 # Trace retransmits for a subnet kubectl retina bpftrace <node> --retransmits --cidr 10.244.0.0/16 ``` ## What's Implemented ### New CLI Command: `kubectl retina bpftrace <node-name>` Traces network issues on a specified Kubernetes node with the following capabilities: **Event Types Captured:** | Type | Probe | Description | |------|-------|-------------| | DROP | `kfree_skb` | Packet drops with kernel reason codes (e.g., NETFILTER_DROP for NetworkPolicy) | | RST_SENT | `tcp_send_reset` | TCP RST packets sent by this host | | RST_RECV | `tcp_receive_reset` | TCP RST packets received by this host | | SOCK_ERR | `inet_sk_error_report` | Socket errors (ECONNREFUSED, ETIMEDOUT, etc.) | | RETRANS | `tcp_retransmit_skb` | TCP retransmissions indicating packet loss | ## Flags | Flag | Description | |------|-------------| | `--ip` | Filter by IP address (src or dst) | | `--cidr` | Filter by CIDR (src or dst) | | `--drops` | Enable only packet drop events | | `--rst` | Enable only TCP RST events | | `--errors` | Enable only socket error events | | `--retransmits` | Enable only retransmit events | | `--all` | Enable all events (default) | | `--duration` | Trace duration (0 = until Ctrl-C) | | `--startup-timeout` | Pod startup timeout | | `-o, --output` | Output format: `table` or `json` | When no event flags are specified, all events are traced. ## Related Issue If this pull request is related to any issue, please mention it here. Additionally, make sure that the issue is assigned to you before submitting this pull request. ## Checklist - [ ] I have read the [contributing documentation](https://retina.sh/docs/Contributing/overview). - [ ] I signed and signed-off the commits (`git commit -S -s ...`). See [this documentation](https://docs.github.com/en/authentication/managing-commit-signature-verification/about-commit-signature-verification) on signing commits. - [ ] I have correctly attributed the author(s) of the code. - [ ] I have tested the changes locally. - [ ] I have followed the project's style guidelines. - [ ] I have updated the documentation, if necessary. - [ ] I have added tests, if applicable. ## Screenshots (if applicable) or Testing Completed <img width="956" height="893" alt="image" src="https://github.com/user-attachments/assets/cfefd21a-03a7-4518-88c5-66a28d3a4145" /> ## Additional Notes ## Limitations - **IPv4 only**: IPv6 not supported - **Linux only**: Windows nodes not supported - **Cilium CNI**: DROP events won't capture Cilium policy drops (Cilium uses eBPF datapath, not netfilter/kfree_skb) ## Testing ```bash # Build go build -o kubectl-retina ./cli # E2E test (validates all 4 event types) ./test/e2e/test_bpftrace_drops.sh # Manual NODE=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}') ./kubectl-retina bpftrace $NODE --duration 30s --retina-shell-image-version v1.0.3 ``` ## Security - IP/CIDR inputs validated and converted to hex (injection-safe) - Commands executed via array-based exec (no shell) - Pod uses minimal capabilities for bpftrace --- Please refer to the [CONTRIBUTING.md](../CONTRIBUTING.md) file for more information on how to contribute to this project. --------- Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Summary
New
bpftracesubcommand for real-time tracing of network issues on Kubernetes nodes using eBPF/bpftrace.Use Cases
What's Implemented
New CLI Command:
kubectl retina bpftrace <node-name>Traces network issues on a specified Kubernetes node with the following capabilities:
Event Types Captured:
kfree_skbtcp_send_resettcp_receive_resetinet_sk_error_reporttcp_retransmit_skbFlags
--ip--cidr--drops--rst--errors--retransmits--all--duration--startup-timeout-o, --outputtableorjsonWhen no event flags are specified, all events are traced.
Related Issue
If this pull request is related to any issue, please mention it here. Additionally, make sure that the issue is assigned to you before submitting this pull request.
Checklist
git commit -S -s ...). See this documentation on signing commits.Screenshots (if applicable) or Testing Completed
Additional Notes
Limitations
Testing
Security
Please refer to the CONTRIBUTING.md file for more information on how to contribute to this project.