Retained ELF string table causes 100+ MiB heap growth per instrumented binary

[bourbonkk]

⚠️ UPDATE 2026-07-13 — root cause re-diagnosed, see correction comment below

The original report below framed the issue as retention on Rust/C++/Envoy binaries, and attributed it to .strtab substring semantics. Both framings are now known to be inaccurate:

• Substring theory is wrong: Go stdlib debug/elf.getString() returns via string([]byte) which always allocates a new backing array. The original .strtab slice is GC-eligible; substrings don't hold it.
• The heap the profiles captured is a mix of two independent causes:
  1. Retained heap on Go binaries — due to a missing probe.Skip = true in pkg/ebpf/instrumenter.go:gatherGoOffsets, which causes link.Executable.Uprobe to be called with Address=0, forcing cilium's lazyLoadSymbols() to parse the full ELF and retain it on Executable.cachedSymbols for the tracer's lifetime.
  2. Transient parse cost when generic tracer's resolveInstrPath falls back to the executable itself (Rust with debug info, Envoy static binary). This is GC-eligible after gatherOffsets returns, but pprof snapshots taken before the next GC show it in inuse-space.

The two clusters in the original report both had Go services co-located (flagd-build, product-catalog in Cluster A; flagd sidecars in Cluster B), so the retained portion of the observed heap was very likely from those, not from the Rust/Envoy processes named in the headline.

PR #2642 fixes cause (1) — retention on Go binaries. Cause (2) (transient parse cost on native-path exe fallback) remains open; it may warrant a separate issue.

The rest of this body is preserved for history but its root-cause section should be read alongside the correction comment.

---

Summary

When instrumenting large binaries (Rust, C++/Envoy, Erlang), OBI holds 100+ MiB of parsed ELF string table per unique instrumented binary in its resident heap. The retention persists after uprobe attach completes, and is not reclaimed by GC until all processes referencing that binary exit.

Root cause appears to be that symbol name strings — created via substring of the raw .strtab []byte — retain a reference to the full slice (Go semantics: string(b[i:j]) does not copy). The parsed link.(*Executable) (containing the symbol name map) is kept alive by existingTracers[Ino()] = tracer in pkg/appolly/discover/attacher.go, so the strtab buffer stays reachable through the substring references indefinitely.

Environment

• OBI version: v0.10.0 (ghcr.io/open-telemetry/opentelemetry-ebpf-instrumentation/ebpf-instrument)
• Deployment: Kubernetes DaemonSet (2 clusters, ~7 nodes each)
• Preset: application (HTTP/gRPC uprobes), network.enable: false
• k8sCache.replicas: 0 (in-agent kube metadata)
• Kernel: Linux 5.15+
• Continuous profiling via Alloy pyroscope.scrape on OBI's profile_port: 6060

Observation

Pyroscope heap inuse (memory:inuse_space) profiles taken from two heavy OBI DaemonSet pods on different clusters show the same dominant stack:

attacherLoop
 └─ traceAttacher.getTracer
    └─ ProcessTracer.NewExecutable
       └─ cilium/ebpf/link.(*Executable).Uprobe
          └─ (*Executable).address
             └─ debug/elf.(*File).Symbols
                └─ debug/elf.(*File).getSymbols64
                   └─ debug/elf.getString                (120–143 MiB self)

Cluster A — single Rust binary

Node has one Rust service (shipping from the OpenTelemetry Demo) plus a few Erlang/Go processes. Heap flame graph:

Symbol	Self	Total
debug/elf.getString	143 MiB	143 MiB
cilium/ebpf/link.(*Executable).load	53.9 MiB	166 MiB

obi_instrumented_processes on that node: shipping (Rust), beam.smp, epmd, erl_child_setup, flagd-build, product-catalog.

Cluster B — two identical Envoy pods

Node runs frontend-proxy × 2 (Envoy, same image → same inode → correctly deduped to 1× tracer via existingTracers[Ino]), plus JVM/Go/Python/C daemon processes.

Total heap on this node ~350 MiB (baseline nodes without large binaries ~250 MiB). Heap breakdown:

Symbol	Self	Total
debug/elf.getString	120 MiB	120 MiB
link.(*Executable).address.func1 (attach path)	0	151 MiB
ringBufForwarder.bgFlushOnTimeout	0	128 MiB

The debug/elf.getString figure is retained long after attach completed (steady state; not transient during startup).

Baseline comparison

Baseline pods (nodes with no Rust/C++/Envoy binaries — only Go/Python/JVM services) show:

• procs.envStrsToMap — ~23 MiB
• k8s informer / cache — ~18 MiB
• no elf.getString / getSymbols64 frames at all in inuse-space

Heap sits around ~250 MiB steady state, versus ~350 MiB on the heavy nodes.

Reproduction

1. Deploy OBI v0.10.0 as DaemonSet with preset: application, internal_metrics.exporter: otel, profile_port: 6060.
2. Ensure at least one Rust or C++ (Envoy) workload is running on one of the nodes.
3. Wait for OBI to complete attacherLoop (a few minutes).
4. curl http://<obi-pod>:6060/debug/pprof/heap | go tool pprof -top -inuse_space -.
5. debug/elf.(*File).getSymbols64 / debug/elf.getString dominates the inuse-space profile.

Suggested fix

The parsed link.(*Executable) no longer needs the strtab byte slice once probe attachment is done — only the {function_name → address} mapping is used at runtime. Two possible directions:

Option 1: Detach symbol strings after attach

Copy the small set of retained symbol names to independent allocations so the underlying strtab slice becomes GC-eligible.

// After successful Uprobe attach in cilium/ebpf/link:
for name, sym := range e.symbols {
    e.symbols[name] = symbol{
        Name: strings.Clone(sym.Name),
        Addr: sym.Addr,
    }
}
// or, at OBI's side, after (*Executable).Uprobe returns,
// release the reference to `e` and keep only what OBI needs.

Option 2: Extract minimal {name, addr} set at parse time

Instead of storing *link.Executable (which transitively retains strtab), OBI's existingTracers could store a compact struct containing only the resolved probe attach points. Executable can be closed / dereferenced after NewExecutable returns.

Expected impact

• 100+ MiB → a few KiB per unique large-symbol binary.
• Reduces steady-state memory requirement for OBI DaemonSet on nodes hosting Rust/C++/Envoy workloads.
• Improves the effective density limit for OBI in production clusters where memory limits are set to accommodate this steady-state footprint.

Related issues

• Support batched/sequential eBPF program loading to reduce startup memory spike #1340 — Startup memory spike from concurrent eBPF program loading. Related but distinct: that issue is the transient startup peak; this one is post-attach steady-state retention.
• fix: Stale inode-keyed cache causing SIGSEGV on binary replacement #1683 — Stale inode-keyed cache causing SIGSEGV. Same existingTracers[Ino] dedup path we reference here.
• Refactor FileInfo onwership semantics #2125 — FileInfo ownership refactor. Adjacent code area.

Happy to provide additional profiles, run experiments, or draft a PR if the proposed direction is acceptable.

[bourbonkk]

Root cause correction

After further investigation, the substring-retention theory in the original report is incorrect. Go stdlib debug/elf.getString() returns via string(section[start:end]), and string([]byte) conversion in Go always allocates a new backing array — the resulting string does not retain a reference to the original strtab []byte. So the strtab slice itself is not the retention culprit.

Actual root cause

The retained heap comes from cilium/ebpf's link.Executable.cachedSymbols map[string]symbol, populated by lazyLoadSymbols() (uprobe.go:172). Once populated, the map is retained for the lifetime of the *Executable, which OBI stores on instrumenter.exe (referenced from ProcessTracer.Instrumentables[ino]). So the symbol map lives as long as the tracer, which is as long as any process for that binary is alive.

The interesting question is why lazyLoadSymbols() is entered at all — OBI is careful to pass UprobeOptions.Address on every call site (pkg/ebpf/instrumenter.go:580, 623, 645), and cilium's address() short-circuits when Address > 0:

// cilium/ebpf link/uprobe.go:196-199
func (ex *Executable) address(symbol string, address, offset uint64) (uint64, error) {
    if address > 0 {
        return address + offset, nil
    }
    err := ex.lazyLoadSymbols()
    ...
}

So the parse is only triggered when OBI ends up calling .Uprobe("", prog, &UprobeOptions{Address: 0}). That happens through this exact path:

pkg/ebpf/instrumenter.go:983 gatherGoOffsets() misses probe.Skip = true on unresolved symbols.

func (i *instrumenter) gatherGoOffsets(goProbes map[string][]*ebpfcommon.ProbeDesc) {
    for symbolName, descs := range goProbes {
        offs, ok := i.offsets.Funcs[symbolName]
        if !ok {
            // the program function is not in the detected offsets. Ignoring
            log.Debug("ignoring function", "function", symbolName)
            continue                          // ← probe.Skip is NOT set
        }
        for _, probe := range descs {
            probe.StartOffset = offs.Start
            probe.ReturnOffsets = offs.Returns
        }
    }
}

Compare to the native path, gatherOffsetsImpl() at line 900-916, which explicitly sets probe.Skip = true when a symbol lookup fails. The asymmetry between the two functions is the bug.

Downstream effect: instrumentProbes() iterates all probes; since probe.Skip == false (default zero value), it calls i.uprobe(exe, probe), which in turn calls exe.Uprobe("", probe.Start, &link.UprobeOptions{Address: probe.StartOffset}). Because probe.StartOffset is still the zero value (never assigned when lookup failed), Address == 0 reaches cilium, which enters lazyLoadSymbols() and parses the entire ELF symbol table into cachedSymbols.

This reproduces reliably on Go binaries because gotracer's GoProbes() (pkg/internal/ebpf/gotracer/gotracer.go:495-...) registers ~60 probes covering net/http, net/rpc/jsonrpc, github.com/lib/pq, golang.org/x/net/http2, crypto/tls, database/sql, etc. Most Go services do not use every one of these libraries, so at least one lookup fails, and cilium ends up parsing the full symbol table for the executable.

Rust / C++ / Envoy case is a separate path

Rust and C++/Envoy processes hit svc.InstrumentableGeneric → generic tracer → GoProbes() == nil, so gatherGoOffsets never runs for them. They instead go through i.uprobes() → uprobeModules() → resolveInstrPath(), which falls back to the executable itself when a library (libssl.so, nginx, libpython3., libjvm.so, etc.) is not found in the process's shared library mappings. gatherOffsets() is then called on the executable path, which does elf.Open + f.Symbols() + f.DynamicSymbols() on the full binary.

For this native path gatherOffsetsImpl does set probe.Skip = true on unresolved symbols, so cilium's lazyLoadSymbols is not entered. The heap cost here is the transient parse itself — Envoy's static binary and Rust with debug info produce very large symbol tables, and although the slice is GC-eligible after gatherOffsets returns, a pprof snapshot captured mid-parse (or before the next GC cycle) can show tens or hundreds of MiB in getString.

So the two clusters exhibit two related but distinct symptoms:

• Cluster A (Rust shipping) — debug/elf.getString: 143 MiB self: mixture of native path parse + retained Go path probes from flagd-build, product-catalog (Go binaries on the same node) whose gotracer probes hit the probe.Skip bug.
• Cluster B (Envoy frontend-proxy × 2) — debug/elf.getString: 120 MiB self: dominated by the native parse of the Envoy binary, plus Go/JVM/Python service tracers on the same node.

Suggested fix

Bring gatherGoOffsets into symmetry with gatherOffsetsImpl:

func (i *instrumenter) gatherGoOffsets(goProbes map[string][]*ebpfcommon.ProbeDesc) {
    log := ilog().With("probes", "gatherGoOffsets")
    for symbolName, descs := range goProbes {
        offs, ok := i.offsets.Funcs[symbolName]
        if !ok {
            log.Debug("ignoring function", "function", symbolName)
            for _, probe := range descs {
                probe.Skip = true
            }
            continue
        }
        for _, probe := range descs {
            probe.Skip = false
            probe.StartOffset = offs.Start
            probe.ReturnOffsets = offs.Returns
        }
    }
}

Effect: probes with unresolved offsets are skipped in instrumentProbes, cilium's lazyLoadSymbols is never entered for these attach attempts, and Executable.cachedSymbols remains empty. Steady-state retention drops accordingly.

Happy to open a PR with this fix and a regression test that verifies probe.Skip is set on missing Go symbols. Would also welcome guidance on whether a companion mitigation for the native-path transient allocation is worth pursuing (e.g. capping / avoiding the exe-fallback in resolveInstrPath when no UProbes() library is found on the process).

[bourbonkk]

Empirical validation (2026-07-13)

Ran the natural-experiment check today. Result strongly confirms the diagnosis in the correction comment above and validates PR #2642.

Setup

Cluster: otel-demo-ay (Kubernetes, 7-node DaemonSet). The heaviest node (worker-bvnh) runs the following instrumented processes:

Process	Language	Instrumented? (before)
shipping	Rust (large symbol table, debug info)	✅
beam.smp + epmd + erl_child_setup	Erlang	✅
flagd (2 sidecar containers)	Go	✅
product-catalog	Go	✅

Baseline OBI DaemonSet pod working_set: ~350 MiB on this node. Baseline nodes (Go-app-less): ~250 MiB.

Experiment

Added flagd and product-catalog to discovery.exclude_instrument in the OBI helm values. Rust shipping and Erlang processes were left instrumented.

Result

Node	Before	After (Go excluded)	Δ
worker-bvnh (Rust + Erlang + Go)	350 MiB	218 MiB	−132 MiB
worker-x644 (Go × 2 + JVM + Node + Python)	~300+ MiB	219 MiB	~−80 MiB

Both nodes dropped to baseline or below, even though the Rust binary and other large native binaries remained instrumented on those nodes.

Cross-check with natural experiment

The otel-demo-ay cluster already has one node (worker-q6jj) with Rust shipping present but no Go apps. That node has always been at baseline (~250 MiB). Combined with the experiment above, this makes the causal story unambiguous:

• Rust + Go apps together → heavy (350 MiB)
• Rust alone → baseline (250 MiB)
• Rust + Go apps, Go apps excluded from OBI → baseline (218 MiB)

The retained heap is a function of instrumented Go apps, not Rust/C++/Envoy binaries as originally framed.

Implication for #2642

Expected per-Go-app savings on nodes with the bug: ~40–70 MiB per unique Go binary. On dense nodes co-hosting many Go services this is dominant. The fix in #2642 makes gatherGoOffsets symmetric with gatherOffsetsImpl so cilium's lazyLoadSymbols is not entered on missed-symbol probes.

Happy to run further validation (e.g. before/after with the actual #2642 patch applied instead of exclude_instrument) once the PR is close to merge.

[grcevski]

Thanks for reporting this @bourbonkk ! Would you like to submit a patch for this issue, we'd gladly accept it. If not one of us will do the fix.

[bourbonkk]

Thanks! Already opened #2642 — appreciate a review when you have time. Root cause analysis and validation numbers are in the follow-up comments above.