Skip to content

feat(cli): Add kubectl retina bpftrace command for real-time network issue tracing - #2061

Merged
Alex Castilio (alexcastilio) merged 21 commits into
mainfrom
feat/drop-trace
Feb 25, 2026
Merged

feat(cli): Add kubectl retina bpftrace command for real-time network issue tracing#2061
Alex Castilio (alexcastilio) merged 21 commits into
mainfrom
feat/drop-trace

Conversation

@alexcastilio

@alexcastilio Alex Castilio (alexcastilio) commented Feb 13, 2026

Copy link
Copy Markdown
Contributor

Description

Summary

New bpftrace subcommand for real-time tracing of network issues on Kubernetes nodes using eBPF/bpftrace.

Use Cases

  • Debug packet drops on a node (e.g., NetworkPolicy blocks, routing issues)
  • Trace TCP connection failures (RST sent/received, connection refused)
  • Identify retransmissions indicating packet loss or congestion
  • Filter events by IP or subnet to focus on specific endpoints
# Trace all events (default)
kubectl retina bpftrace <node>

# Trace only drops and RSTs for a specific IP
kubectl retina bpftrace <node> --drops --rst --ip 10.244.1.15

# Trace retransmits for a subnet
kubectl retina bpftrace <node> --retransmits --cidr 10.244.0.0/16

What's Implemented

New CLI Command: kubectl retina bpftrace <node-name>

Traces network issues on a specified Kubernetes node with the following capabilities:

Event Types Captured:

Type Probe Description
DROP kfree_skb Packet drops with kernel reason codes (e.g., NETFILTER_DROP for NetworkPolicy)
RST_SENT tcp_send_reset TCP RST packets sent by this host
RST_RECV tcp_receive_reset TCP RST packets received by this host
SOCK_ERR inet_sk_error_report Socket errors (ECONNREFUSED, ETIMEDOUT, etc.)
RETRANS tcp_retransmit_skb TCP retransmissions indicating packet loss

Flags

Flag Description
--ip Filter by IP address (src or dst)
--cidr Filter by CIDR (src or dst)
--drops Enable only packet drop events
--rst Enable only TCP RST events
--errors Enable only socket error events
--retransmits Enable only retransmit events
--all Enable all events (default)
--duration Trace duration (0 = until Ctrl-C)
--startup-timeout Pod startup timeout
-o, --output Output format: table or json

When no event flags are specified, all events are traced.

Related Issue

If this pull request is related to any issue, please mention it here. Additionally, make sure that the issue is assigned to you before submitting this pull request.

Checklist

  • I have read the contributing documentation.
  • I signed and signed-off the commits (git commit -S -s ...). See this documentation on signing commits.
  • I have correctly attributed the author(s) of the code.
  • I have tested the changes locally.
  • I have followed the project's style guidelines.
  • I have updated the documentation, if necessary.
  • I have added tests, if applicable.

Screenshots (if applicable) or Testing Completed

image

Additional Notes

Limitations

  • IPv4 only: IPv6 not supported
  • Linux only: Windows nodes not supported
  • Cilium CNI: DROP events won't capture Cilium policy drops (Cilium uses eBPF datapath, not netfilter/kfree_skb)

Testing

# Build
go build -o kubectl-retina ./cli

# E2E test (validates all 4 event types)
./test/e2e/test_bpftrace_drops.sh

# Manual
NODE=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')
./kubectl-retina bpftrace $NODE --duration 30s --retina-shell-image-version v1.0.3

Security

  • IP/CIDR inputs validated and converted to hex (injection-safe)
  • Commands executed via array-based exec (no shell)
  • Pod uses minimal capabilities for bpftrace

Please refer to the CONTRIBUTING.md file for more information on how to contribute to this project.

@github-actions

github-actions Bot commented Feb 16, 2026

Copy link
Copy Markdown

Retina Code Coverage Report

Total coverage increased from 33.1% to 33.9%

Increased diff

Impacted Files Coverage
cli/cmd/config.go 0.0% ... 100.0% (100.0%) ⬆️
cli/cmd/trace.go 0.0% ... 100.0% (100.0%) ⬆️
cli/cmd/version.go 0.0% ... 100.0% (100.0%) ⬆️
cli/cmd/shell.go 0.0% ... 65.2% (65.2%) ⬆️
cli/cmd/root.go 0.0% ... 100.0% (100.0%) ⬆️

@alexcastilio Alex Castilio (alexcastilio) changed the title feat(shell): Add nettrace feat(cli): Add kubectl retina bpftrace command for real-time network issue tracing Feb 16, 2026
Comment thread test/e2e/test_bpftrace_drops.sh
Comment thread test/e2e/test_bpftrace_drops.sh Outdated

@SRodi Simone Rodigari (SRodi) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current kfree_skb logic parses every single dropped packet (including invalid ones, or ones valid but not matching filter) to extracting IPs before filtering, which is expensive. Using predicate filtering would be more efficient.

Comment thread test/e2e/test_bpftrace_drops.sh
Comment thread cli/cmd/bpftrace.go
Comment thread cli/cmd/bpftrace.go Outdated
Comment thread cli/cmd/bpftrace.go
Comment thread cli/cmd/bpftrace.go
Comment thread cli/cmd/bpftrace.go

@SRodi Simone Rodigari (SRodi) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

…dation

Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
…anagement

Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
- ScriptGenerator creates bpftrace script for kfree_skb tracepoint
- DropReasonsCommand() fetches drop reason enum from kernel at runtime
- Outputs numeric reason codes (kernel-version specific)
- Hex conversion for IP/CIDR filters prevents injection
- Supports both table and JSON output formats
- Use Args (not Command) to preserve entrypoint.sh debugfs mounting
- Add required capabilities: MKNOD, SYS_CHROOT (per shell.md)
- Add AppArmor unconfined profile for bpftrace
- Handle duration timeout gracefully

Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
…el retransmit reason code

Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
…obe-specific semantics

Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
…nt selection (--drops, --rst, --errors, --retransmits)

Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
…ive in test result

Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
…ssage

Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
…UEUE drops with no consumer

Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
…ignal.Stop cleanup

Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an experimental kubectl retina bpftrace command to run bpftrace-based network issue tracing on a target Kubernetes node (via a host-network debug pod), with optional IP/CIDR filtering and table/JSON output.

Changes:

  • Introduces bpftrace CLI subcommand with flags for event selection, filtering, output format, and duration/startup timeout.
  • Implements bpftrace script generation (drops/RST/errors/retransmits/NFQUEUE) plus typed validation and unit tests.
  • Adds troubleshooting documentation and E2E shell scripts for basic coverage and IP-filter behavior.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 14 comments.

Show a summary per file
File Description
cli/cmd/bpftrace.go New bpftrace Cobra command, flag parsing, validation, and trace invocation.
cli/cmd/bpftrace_test.go Unit tests for IP/CIDR/output validation and injection-pattern rejection.
shell/trace.go Trace execution on a node via a host-network pod, capability setup, and exec streaming.
shell/trace_test.go Unit tests for capabilities, TraceConfig typing, pod security context, and exec patterns.
shell/tracescript.go bpftrace script generator, including injection-safe IP/CIDR filtering logic.
shell/tracescript_test.go Unit tests for script contents, JSON/table output, and filter generation correctness.
docs/06-Troubleshooting/bpftrace.md User-facing docs for the new command, event types, flags, and output examples.
test/e2e/test_bpftrace_drops.sh E2E script to generate representative events and validate capture presence.
test/e2e/test_bpftrace_filter_ip.sh E2E script validating --ip filtering correctness across event types.
cli/README.md Mentions the new bpftrace subcommand in CLI README.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread cli/cmd/bpftrace.go
Comment thread cli/cmd/bpftrace.go
Comment thread shell/trace.go
Comment thread shell/trace.go
Comment thread cli/cmd/bpftrace.go
Comment thread test/e2e/test_bpftrace_filter_ip.sh
Comment thread test/e2e/test_bpftrace_filter_ip.sh
Comment thread test/e2e/test_bpftrace_filter_ip.sh
Comment thread test/e2e/test_bpftrace_drops.sh
Comment thread test/e2e/test_bpftrace_drops.sh
@alexcastilio
Alex Castilio (alexcastilio) added this pull request to the merge queue Feb 25, 2026
Merged via the queue into main with commit 927be7b Feb 25, 2026
35 of 36 checks passed
@alexcastilio
Alex Castilio (alexcastilio) deleted the feat/drop-trace branch February 25, 2026 16:32
Laksh (lakshk98) pushed a commit to lakshk98/retina that referenced this pull request Aug 4, 2026
…k issue tracing (microsoft#2061)

# Description

## Summary

New `bpftrace` subcommand for real-time tracing of network issues on
Kubernetes nodes using eBPF/bpftrace.

## Use Cases

- Debug packet drops on a node (e.g., NetworkPolicy blocks, routing
issues)
- Trace TCP connection failures (RST sent/received, connection refused)
- Identify retransmissions indicating packet loss or congestion
- Filter events by IP or subnet to focus on specific endpoints

```bash
# Trace all events (default)
kubectl retina bpftrace <node>

# Trace only drops and RSTs for a specific IP
kubectl retina bpftrace <node> --drops --rst --ip 10.244.1.15

# Trace retransmits for a subnet
kubectl retina bpftrace <node> --retransmits --cidr 10.244.0.0/16
```

## What's Implemented

### New CLI Command: `kubectl retina bpftrace <node-name>`

Traces network issues on a specified Kubernetes node with the following
capabilities:

**Event Types Captured:**
| Type | Probe | Description |
|------|-------|-------------|
| DROP | `kfree_skb` | Packet drops with kernel reason codes (e.g.,
NETFILTER_DROP for NetworkPolicy) |
| RST_SENT | `tcp_send_reset` | TCP RST packets sent by this host |
| RST_RECV | `tcp_receive_reset` | TCP RST packets received by this host
|
| SOCK_ERR | `inet_sk_error_report` | Socket errors (ECONNREFUSED,
ETIMEDOUT, etc.) |
| RETRANS | `tcp_retransmit_skb` | TCP retransmissions indicating packet
loss |

## Flags

| Flag | Description |
|------|-------------|
| `--ip` | Filter by IP address (src or dst) |
| `--cidr` | Filter by CIDR (src or dst) |
| `--drops` | Enable only packet drop events |
| `--rst` | Enable only TCP RST events |
| `--errors` | Enable only socket error events |
| `--retransmits` | Enable only retransmit events |
| `--all` | Enable all events (default) |
| `--duration` | Trace duration (0 = until Ctrl-C) |
| `--startup-timeout` | Pod startup timeout |
| `-o, --output` | Output format: `table` or `json` |

When no event flags are specified, all events are traced.

## Related Issue

If this pull request is related to any issue, please mention it here.
Additionally, make sure that the issue is assigned to you before
submitting this pull request.

## Checklist

- [ ] I have read the [contributing
documentation](https://retina.sh/docs/Contributing/overview).
- [ ] I signed and signed-off the commits (`git commit -S -s ...`). See
[this
documentation](https://docs.github.com/en/authentication/managing-commit-signature-verification/about-commit-signature-verification)
on signing commits.
- [ ] I have correctly attributed the author(s) of the code.
- [ ] I have tested the changes locally.
- [ ] I have followed the project's style guidelines.
- [ ] I have updated the documentation, if necessary.
- [ ] I have added tests, if applicable.

## Screenshots (if applicable) or Testing Completed

<img width="956" height="893" alt="image"
src="https://github.com/user-attachments/assets/cfefd21a-03a7-4518-88c5-66a28d3a4145"
/>


## Additional Notes

## Limitations

- **IPv4 only**: IPv6 not supported
- **Linux only**: Windows nodes not supported
- **Cilium CNI**: DROP events won't capture Cilium policy drops (Cilium
uses eBPF datapath, not netfilter/kfree_skb)

## Testing

```bash
# Build
go build -o kubectl-retina ./cli

# E2E test (validates all 4 event types)
./test/e2e/test_bpftrace_drops.sh

# Manual
NODE=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')
./kubectl-retina bpftrace $NODE --duration 30s --retina-shell-image-version v1.0.3
```

## Security

- IP/CIDR inputs validated and converted to hex (injection-safe)
- Commands executed via array-based exec (no shell)
- Pod uses minimal capabilities for bpftrace

---

Please refer to the [CONTRIBUTING.md](../CONTRIBUTING.md) file for more
information on how to contribute to this project.

---------

Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Laksh (lakshk98) pushed a commit to lakshk98/retina that referenced this pull request Aug 4, 2026
…k issue tracing (microsoft#2061)

# Description

## Summary

New `bpftrace` subcommand for real-time tracing of network issues on
Kubernetes nodes using eBPF/bpftrace.

## Use Cases

- Debug packet drops on a node (e.g., NetworkPolicy blocks, routing
issues)
- Trace TCP connection failures (RST sent/received, connection refused)
- Identify retransmissions indicating packet loss or congestion
- Filter events by IP or subnet to focus on specific endpoints

```bash
# Trace all events (default)
kubectl retina bpftrace <node>

# Trace only drops and RSTs for a specific IP
kubectl retina bpftrace <node> --drops --rst --ip 10.244.1.15

# Trace retransmits for a subnet
kubectl retina bpftrace <node> --retransmits --cidr 10.244.0.0/16
```

## What's Implemented

### New CLI Command: `kubectl retina bpftrace <node-name>`

Traces network issues on a specified Kubernetes node with the following
capabilities:

**Event Types Captured:**
| Type | Probe | Description |
|------|-------|-------------|
| DROP | `kfree_skb` | Packet drops with kernel reason codes (e.g.,
NETFILTER_DROP for NetworkPolicy) |
| RST_SENT | `tcp_send_reset` | TCP RST packets sent by this host |
| RST_RECV | `tcp_receive_reset` | TCP RST packets received by this host
|
| SOCK_ERR | `inet_sk_error_report` | Socket errors (ECONNREFUSED,
ETIMEDOUT, etc.) |
| RETRANS | `tcp_retransmit_skb` | TCP retransmissions indicating packet
loss |

## Flags

| Flag | Description |
|------|-------------|
| `--ip` | Filter by IP address (src or dst) |
| `--cidr` | Filter by CIDR (src or dst) |
| `--drops` | Enable only packet drop events |
| `--rst` | Enable only TCP RST events |
| `--errors` | Enable only socket error events |
| `--retransmits` | Enable only retransmit events |
| `--all` | Enable all events (default) |
| `--duration` | Trace duration (0 = until Ctrl-C) |
| `--startup-timeout` | Pod startup timeout |
| `-o, --output` | Output format: `table` or `json` |

When no event flags are specified, all events are traced.

## Related Issue

If this pull request is related to any issue, please mention it here.
Additionally, make sure that the issue is assigned to you before
submitting this pull request.

## Checklist

- [ ] I have read the [contributing
documentation](https://retina.sh/docs/Contributing/overview).
- [ ] I signed and signed-off the commits (`git commit -S -s ...`). See
[this
documentation](https://docs.github.com/en/authentication/managing-commit-signature-verification/about-commit-signature-verification)
on signing commits.
- [ ] I have correctly attributed the author(s) of the code.
- [ ] I have tested the changes locally.
- [ ] I have followed the project's style guidelines.
- [ ] I have updated the documentation, if necessary.
- [ ] I have added tests, if applicable.

## Screenshots (if applicable) or Testing Completed

<img width="956" height="893" alt="image"
src="https://github.com/user-attachments/assets/cfefd21a-03a7-4518-88c5-66a28d3a4145"
/>


## Additional Notes

## Limitations

- **IPv4 only**: IPv6 not supported
- **Linux only**: Windows nodes not supported
- **Cilium CNI**: DROP events won't capture Cilium policy drops (Cilium
uses eBPF datapath, not netfilter/kfree_skb)

## Testing

```bash
# Build
go build -o kubectl-retina ./cli

# E2E test (validates all 4 event types)
./test/e2e/test_bpftrace_drops.sh

# Manual
NODE=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')
./kubectl-retina bpftrace $NODE --duration 30s --retina-shell-image-version v1.0.3
```

## Security

- IP/CIDR inputs validated and converted to hex (injection-safe)
- Commands executed via array-based exec (no shell)
- Pod uses minimal capabilities for bpftrace

---

Please refer to the [CONTRIBUTING.md](../CONTRIBUTING.md) file for more
information on how to contribute to this project.

---------

Signed-off-by: Alex Castilio dos Santos <alexsantos@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants