ziti-edge-tunnel 1.18.1 on ARM64 retains ~1 GiB after media streams and eventually OOMs

[jsbrain]

Summary

ziti-edge-tunnel 1.18.1 on ARM64 reproducibly retains a large anonymous/data allocation after forwarding Jellyfin media traffic.

A clean process starts at approximately 10.5 MiB container memory and 52.9 MiB VmData. After a multi-stream workload with a short additional phone stream (likely transcoded), it holds approximately 1.015 GiB container memory and 1.104 GiB VmData. When every stream is stopped, CPU returns to idle and the large media socket disappears, but the allocation remains unchanged.

During longer operation, combined tunnel RSS and swap reached approximately 4.08 GiB on a VM with 3.7 GiB RAM and 4 GiB swap. Linux has repeatedly OOM-killed the tunneler at approximately 3.5 GiB anonymous RSS, temporarily taking down every service using that endpoint.

The same workload was subsequently run through a full ziti-router 2.0.0 using the Go TPROXY tunneler on the same host and network. With five simultaneous streams, that router peaked below 100 MiB working memory and released the active allocation to approximately 45.7 MiB after the streams stopped. This suggests the retained allocation is specific to the C endpoint-tunneler path rather than Jellyfin, Nginx, Docker, or the host.

Environment

• Architecture: aarch64, ARM Neoverse-N1, 2 vCPU
• OS: Ubuntu 24.04.4 LTS
• Kernel: 6.8.0-134-generic (PREEMPT_DYNAMIC)
• RAM: 3.7 GiB
• Swap: 4 GiB
• Docker client: 23.0.3
• Docker server: 24.0.7
• Image: openziti/ziti-edge-tunnel:1.18.1
• Image architecture: arm64
• Image ID: sha256:08746c4f567bee268dede3aa8474feac4a7baa1626f62b602f847bab188867c4
• Repo digest: openziti/ziti-edge-tunnel@sha256:dcfdab9abd44f51b772feca36832798f49bb3e05ed369b9a08a4d51243eeeb0f
• Ziti C SDK reported by the process: 1.18.1, commit 356a6f7
• Controller and edge routers: 2.0.0, amd64
• Container networking: host
• Container privileges: privileged, with /dev/net/tun
• Restart policy: unless-stopped
• Intercept: private /16, TCP and UDP, ports 1-65535
• Application path: public Nginx Proxy Manager -> local Ziti intercept -> Ziti fabric -> private Jellyfin TCP port 8096

The same operational symptom was previously seen with endpoint image 1.9.8. The controlled measurements below are from 1.18.1.

Current-version caveat

At the time of filing, the latest ziti-tunnel-sdk-c release is 1.18.3. Production was moved to the Go-router mitigation after the OOM incidents, so the complete reproduction has not yet been repeated on 1.18.3.

The ziti-tunnel-sdk-c history from 1.18.1 through 1.18.3 contains two partial-authentication changes and a CI action bump; I did not find an obvious TCP tunneler or memory-management change in that range. I can test a maintainer-recommended 1.18.3/debug build if needed.

Reproduction

1. Restart only the ziti-edge-tunnel container.
2. Wait for controller authorization, both edge-router channels, and intercept routes to become ready.
3. Confirm idle container memory near 10.5 MiB and tunnel swap at zero.
4. Start two Jellyfin streams through the Nginx/Ziti path.
5. Start a third stream from a phone. In this environment that stream appears to trigger a heavier/transcoding path.
6. Observe the tunneler consume approximately one full CPU core and memory rise above 1 GiB.
7. Stop the phone stream, then stop both original streams.
8. Observe CPU return to idle and the media connections close while the approximately 1 GiB allocation remains.

Repeated workloads over longer uptime continue growing the process until the kernel OOM-kills it.

Measurements

Long-running state before controlled restart

After approximately 36 hours:

VmSize:  4592496 kB
VmRSS:   2454776 kB
VmData:  4316596 kB
VmSwap:  1824480 kB

VmRSS + VmSwap was 4,279,256 KiB (approximately 4.08 GiB).

Immediately after restarting only the tunneler

container memory: 10.27 MiB
VmSize:   262596 kB
VmRSS:     15164 kB
VmData:    52292 kB
VmSwap:        0 kB

Within 40 seconds it stabilized around 10.46 MiB container memory, 15.8 MiB RSS, 52.9 MiB VmData, and zero swap.

Idle failed-dial control

There are some removed/offline services in this network. Failed dials (no route to host, connection refused, and no online terminators) continued after restart.

Over approximately two idle minutes, the error-line count increased from 42 to 106 while memory barely changed:

VmRSS:  15812 -> 15872 kB
VmData: 52892 -> 52972 kB
VmSwap: 0 kB throughout

This noisy idle control did not produce the media workload's gigabyte-scale growth.

Active media state

High-frequency sampling recorded:

CPU:              99-104% continuously
container memory: approximately 1.015-1.053 GiB
VmRSS:            approximately 1,067,000 kB
VmData:           approximately 1,104,500 kB
VmSwap:           0 kB
threads:          5

Per-thread sampling showed the main ziti-edge-tunnel/libuv event-loop thread running near 100%, while all four libuv-worker threads were sleeping in futex_wait_queue.

One Nginx-owned intercepted Jellyfin socket showed:

bytes_received: 1,078,456,330
Recv-Q:         2,973,706
Send-Q:         0

The similarity between transported bytes and retained memory is notable, but does not prove that every transported byte was leaked.

After all streams stopped

For more than 30 seconds, all samples were unchanged:

CPU:              0.03-0.26%
container memory: 1.015 GiB
VmSize:           1380204 kB
VmRSS:            1067336 kB
VmData:           1104496 kB
VmSwap:           0 kB

The main thread was sleeping in ep_poll. The large media socket had disappeared. One small keepalive remained with empty queues. No measurable allocation was returned.

Relevant logs around stream termination

ERROR tunnel-sdk:tunnel_tcp.c:191 on_tcp_client_err() client=tcp:<intercept-ip>:<ephemeral-port> err=-14, terminating connection
WARN  ziti-sdk:connect.c:231 close_conn_internal() conn[.../Closed](<media-service>) dumping 2633108 bytes of undelivered data
WARN  ziti-sdk:channel.c:688 dispatch_message() received message without conn_id or for unknown connection ct[DataType] conn_id[680]

The unknown connection warning then repeated many times for connection IDs 679 and 680.

OOM evidence

The kernel recorded at least two OOM kills on 2026-07-13:

19:40:12 UTC: Out of memory: Killed process (ziti-edge-tunnel) total-vm:7899700kB, anon-rss:3566092kB
21:13:37 UTC: Out of memory: Killed process (ziti-edge-tunnel) total-vm:7888892kB, anon-rss:3588536kB

Each kill caused Docker to restart the endpoint and made all proxied services unavailable during reconnection.

Comparative mitigation result: Go router TPROXY

The C endpoint was replaced with openziti/ziti-router:2.0.0, configured with binding: tunnel and mode: tproxy, on the same ARM64 host and Ziti network.

During four HD streams plus a fifth phone stream for approximately 20 seconds:

peak router CPU:        128-161% across the 2-vCPU host
active working memory:  approximately 83-98 MiB
health/restarts/OOM:    healthy / 0 / false

After every stream stopped:

router CPU:             approximately 1-2%
working memory:         stable at 45.7 MiB
process swap:           0
health/restarts/OOM:    healthy / 0 / false

Public Jellyfin health continued returning HTTP 200. This short comparative test did not reproduce the C tunneler's single-core saturation, rapid ~1 GiB growth, or failure to release the active working set.

Expected behavior

• Memory used for forwarded or undelivered media data should be released after delivery or connection termination.
• Sustained stream size should not cause unbounded process growth.
• Late data for a closed connection should be discarded without retaining buffers or leaving the event loop spinning.

Actual behavior

• A fresh endpoint grows by roughly 1 GiB during this media workload.
• The allocation remains after the streams and large media sockets are gone.
• Repeated workloads accumulate enough memory to force swapping and eventually OOM-kill the endpoint.
• During the active failure state, the single main event-loop thread consumes one full CPU core.

Interpretation and evidence boundary

Directly established:

• The retained memory is private process allocation, not filesystem cache or Docker overhead.
• Restarting only the tunneler releases it.
• Idle failed-circuit churn does not explain the observed growth rate.
• Media traffic is associated with full-core event-loop use and gigabyte-scale allocation.
• The allocation persists after media connections and CPU activity cease.
• The equivalent Go-router TPROXY workload remains below 100 MiB and releases most of its active working set.

Measurement limitation:

• The operator directly observed the rapid increase when the additional phone stream started. High-frequency remote sampling began after most of that increase and captured the resulting plateau, thread/socket state, and failure to release memory, rather than every allocation during the initial slope.

Strong hypothesis, not yet source-level proof:

• A buffer/message cleanup path associated with sustained TCP delivery, backpressure, or connection cancellation retains allocations.
• The undelivered data cleanup and subsequent late DataType messages for already-closed connection IDs may be related.

Heap/allocator instrumentation or a maintainer-provided debug build is needed to identify the allocation site.

Questions for maintainers

1. Is this a known issue in the C SDK/tunneler TCP buffer cleanup path, especially following cancellation or backpressure?
2. Could close_conn_internal() dumping undelivered data followed by late DataType messages leave message buffers retained?
3. Is there a recommended ASan/LSan, allocator-profiling, or debug build for capturing the allocation site under this workload?
4. Would you prefer a reproduction on 1.18.3, and are there specific logging or build flags that would make it useful?
5. What additional sanitized circuit/socket metadata would help narrow this down?

Privacy

Public/private addresses, domains, identity material, controller names, circuit IDs, and authentication output have been removed. Exact timestamps and additional sanitized logs can be provided if useful.

[scareything]

Thanks for letting us know. I'll look into this promptly! If there's any way you could run ziti-edge-tunnel under valgrind, and share the xml report that would be super helpful.

[jsbrain]

Thanks — I was able to reproduce this under Valgrind 3.26.0 on ARM64 using the published openziti/ziti-edge-tunnel:1.18.1 image.

During concurrent Jellyfin streaming, the instrumented tunnel stayed at about 100–101% CPU and container memory rose continuously from 566 MiB to 1.683 GiB in 68 seconds. Nginx Proxy Manager remained stable at about 163 MiB. I stopped the test cleanly before the host ran out of memory.

The XML reports 502,134,522 bytes still in use at exit. The dominant record is:

498,460,221 bytes in 48,602 blocks are still reachable

malloc
  conn_inbound_data_msg          ziti-sdk-c/library/connect.c:986
  process_edge_message           ziti-sdk-c/library/connect.c:1555
  flush_to_client                ziti-sdk-c/library/connect.c:922
  on_flush                       ziti-sdk-c/library/connect.c:835
  ztx_process_deadlines          ziti-sdk-c/library/ziti.c:1890
  uv__run_timers

The complete Valgrind XML is attached as a ZIP. The test ran for 10 minutes 11.503 seconds in total, including idle/setup time before the streaming workload. Valgrind options were:

--leak-check=full --show-leak-kinds=all --track-origins=yes
--num-callers=50 --error-limit=no --vgdb=no --xml=yes

Please let me know if you would like another run with different options or a newer/debug build.

valgrind-ziti-edge-tunnel-1.18.1-arm64.zip

[scareything]

Hi,

I haven't been able to reproduce your issue yet, but I did see a potential leak in ziti-sdk-c which is hopefully plugged with openziti/ziti-sdk-c#1088. I'm attaching a build of ziti-edge-tunnel that contains this change here. Could you please run it in your environment and let me know if it fixes the leak you're seeing? Meanwhile I'll continue working on an an environment to reproduce the leak.

linux-arm64-ziti-sdk-c-cb7be02.zip

[jsbrain]

Thanks for the quick build. Unfortunately, cb7be02 does not fix the issue in my environment.

I tested the supplied ARM64 binary directly, without Valgrind. It reports v1.18.3.2; the binary SHA-256 is 4737f8937aec405398e06007696e67bbf91e8e417544eb61855f4df0a8d446ba.

With the same concurrent Jellyfin workload, the behavior reproduced immediately:

UTC       memory       CPU
15:48:05  15.02 MiB    42.49%
15:48:09  128.4 MiB   100.28%
15:48:21  531.9 MiB   100.43%
15:48:34  901 MiB     101.98%
15:48:42  1.126 GiB   100.28%
15:48:50  1.338 GiB   100.29%
15:48:54  1.457 GiB   100.29%

After all streams stopped, memory dropped only partially to 1.279 GiB and then remained unchanged. More than 30 seconds later, CPU was back at 0.05–0.82%, but memory was still 1.279 GiB in every sample.

The patched build also continued to emit a burst of:

received message without conn_id or for unknown connection ct[DataType]

after the media connection closed.

So the close/disconnect race fixed by openziti/ziti-sdk-c#1088 may be real, but it does not appear to be the cause of this retained-buffer issue. The full-core active behavior, rapid traffic-correlated growth, retained memory after idle, and late DataType messages all remain.

I stopped the test before memory pressure became dangerous and restored the Go-router TPROXY mitigation. I can test another instrumented or patched ARM64 build if useful.

[scareything]

I suspect that the "leak" is ultimately caused by ziti-edge-tunnel/lwip being unable to write data to the jellyfin connection when lwip's tcp send buffer shrinks. There was an inadvertent reduction of the send buffer configuration a while ago (when lwip was updated to 2.2.0) - I'm working on correcting that now and should have another build to try shortly.

I'm curious if you were using a previous version of ziti-edge-tunnel that didn't exhibit this problem?

[scareything]

Here's a build of ziti-edge-tunnel that increases the tcp send buffer to a size that should eliminate or at least reduce the need for buffering in the sdk.

linux-arm64-8d8f24e.zip

[jsbrain]

Thanks, I tested the supplied ARM64 (v1.18.3.4) without Valgrind. Unfortunately, increasing TCP_SND_BUF does not fix the issue in my environment.

With the same concurrent Jellyfin workload:

UTC       container memory   CPU       VmData       host available
07:58:54  11 MiB             33.14%     74,068 kB   3,104 MiB
07:59:02  260.1 MiB         100.53%    322,848 kB   2,890 MiB
07:59:14  635.7 MiB         101.29%    712,828 kB   2,517 MiB
07:59:27  904.2 MiB          99.98%    995,296 kB   2,236 MiB
07:59:39  1.233 GiB          99.78%  1,342,228 kB   1,892 MiB
07:59:50  1.802 GiB         100.18%  1,984,720 kB   1,280 MiB
07:59:58  2.530 GiB          99.89%  2,750,364 kB     522 MiB
08:00:02  2.939 GiB         103.84%  3,176,296 kB     123 MiB

The host had no swap for this run. SSH became unresponsive under the memory pressure, so the host was restarted after the test. Docker reports exit code 1 and OOMKilled=false for this run.

The most useful result is the cleanup log. Three media connections reported:

dumping 655984400 bytes of undelivered data
dumping 842440820 bytes of undelivered data
dumping 1877037560 bytes of undelivered data

That is 3,375,462,780 bytes (~3.14 GiB) of undelivered data, which closely matches the measured process VmData of 3,176,296 kB. The build also continued to emit late DataType messages for unknown/closed connection IDs.

So I think your backpressure diagnosis is correct: the Jellyfin/lwIP-side connection is not draining as quickly as data arrives from Ziti. However, doubling the lwIP send buffer does not bound or eliminate the SDK's upstream buffering; the accumulated undelivered data still tracks the process-memory growth almost directly.

Regarding older versions: I saw the same operational symptom with the 1.9.8 endpoint image, but the controlled measurements and source-level evidence were collected on 1.18.1 and these maintainer builds, so I cannot claim the old occurrence had an identical internal cause.

[scareything]

WARN ziti-sdk:connect.c:231 close_conn_internal() conn.../Closed dumping 2633108 bytes of undelivered data

When this message is logged, the buffer that holds the queued inbound data is being freed:

        if (buffer_available(conn->inbound) > 0) {
            CONN_LOG(WARN, "dumping %zd bytes of undelivered data", buffer_available(conn->inbound));
        }
        free_buffer(conn->inbound);

Note that freeing the buffers may not be immediately apparent from looking at the process stats externally.

This data point makes me more interested in understanding why the data is backing up in the first place. It would seem to not be related to the TCP send window but I'm really only guessing at this point. (Also I've still not been able to reproduce this issue myself).

If you're still interesting in pursuing this with me, the next data points that I can think of would be TRACE-level logs from ziti-edge-tunnel, and a packet capture between the tunneler and NPM.

[jsbrain]

While I want to help resolve this to prevent issues for others, my setup is currently working, and I cannot dedicate significant time to troubleshooting.

However, I think I could pretty easily spin up a few temporary environments replicating this setup and provide you full root access for a few days of testing.

Would that be interesting for you?

[scareything]

Thanks for offering access to a reproduction environment! Yes, I'll take you up on that if you can do it. Some time next week would be ideal for me. PTO and other commitments will complicate my schedule for a couple of weeks after that.

[jsbrain]

@scareything sent you an email just now :)