Skip to content

Commit 2d41e08

Browse files
andypostclaude
andcommitted
fix: post-merge review cleanups (proxy keepalive, cgroup dir leak)
Two P2 findings from Codex on freeunitorg#139 (pre-1.35.6 -> master), verified against the code before fixing. nxt_h1proto.c: a duplicate/invalid upstream Content-Length sets r->inconsistent to disable downstream keepalive, but nxt_h1p_peer_closed reassigned the flag from the framing state on the clean upstream close, clearing it -- so an HTTP/1.1 client could reuse the connection past the re-framed response. The flag is overloaded: it also drives dropping the terminal chunk in nxt_h1p_chunk_create (truncation detection, freeunitorg#72). Split the two -- add r->truncated for the terminal-chunk omission and only ever *set* (never clear) r->inconsistent -- so a dup-CL response disables keepalive (the socket is closed at request_close) while keeping its complete terminal chunk. Adds test_proxy_dup_cl_keepalive_disabled asserting the socket close (the Connection header is chosen at header-send time, before keepalive is reduced, so no "Connection: close" is emitted) and the intact terminal chunk; refreshes the now-stale note in test_proxy_dup_cl.py. nxt_cgroup.c: if the pool allocation that caches the created cgroup directory for cleanup failed, setup still moved the process into the cgroup and returned success while nxt_cgroup_cleanup() -- which depends on that cache -- bailed out, leaking the directory under memory pressure. Fail setup and remove the directory it created, including any now-empty parents nxt_fs_mkdir_p() added, via a shared nxt_cgroup_rmdir_up() walk (also used by nxt_cgroup_cleanup). Two latent bugs in the extracted cleanup loop are fixed in the shared helper: nxt_mk_cgpath(task, "", ...) returns a trailing slash that the old strcmp() boundary missed, and an absolute isolation "path" resolves under NXT_CGROUP_ROOT (not under cgroot) so cgroot is never reached -- both let the walk run past cgroot toward the mount root (harmless only because rmdir() fails on the non-empty ancestors). The walk now compares against the slash-trimmed cgroot length and never touches NXT_CGROUP_ROOT or its parents. CHANGES and docs/changes.xml updated under the 1.36.0 block. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017r3PHLb7YSsDTGmuJNpfPm
1 parent dea537f commit 2d41e08

7 files changed

Lines changed: 227 additions & 30 deletions

File tree

CHANGES

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,12 @@ Changes with FreeUnit 1.36.0 16 Jul 2026
4949
body is read to EOF instead of trusting either length, preventing a
5050
response-smuggling desynchronization.
5151

52+
*) Bugfix: keep downstream keepalive disabled after a duplicate or invalid
53+
upstream "Content-Length"; a clean upstream close no longer re-enables
54+
it, so an HTTP/1.1 client cannot reuse the connection past the re-framed
55+
response. The complete body still ends with its terminal chunk — body
56+
truncation is now tracked separately from the inconsistent flag.
57+
5258
*) Bugfix: bound the request-header fields region against the shared
5359
memory message size in libunit, validate every field name/value
5460
serialized pointer and the cached header-field indexes before use,
@@ -195,6 +201,12 @@ Changes with FreeUnit 1.36.0 16 Jul 2026
195201
preserves the original length and fails with ENAMETOOLONG when
196202
"/cgroup.procs" plus the trailing NUL cannot fit.
197203

204+
*) Bugfix: fail cgroup setup when the per-process pool allocation that
205+
caches the created directory for cleanup fails; the directory is now
206+
removed and setup errors out, rather than moving the process into a
207+
cgroup that could never be cleaned up — a cgroup directory leak under
208+
memory pressure.
209+
198210
*) Bugfix: reject a "rootfs" that lexically resolves to "/" (such as
199211
"/.", "/..", or "/foo/.."); chroot("/") is a no-op and would silently
200212
defeat rootfs isolation.

docs/changes.xml

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,25 @@ with ENAMETOOLONG when "/cgroup.procs" plus the trailing NUL cannot fit.
9797
</para>
9898
</change>
9999

100+
<change type="bugfix">
101+
<para>
102+
fail cgroup setup when the per-process pool allocation that caches the
103+
created directory for cleanup fails; the directory is removed and setup
104+
errors out instead of moving the process into a cgroup that could never be
105+
cleaned up -- a cgroup directory leak under memory pressure.
106+
</para>
107+
</change>
108+
109+
<change type="bugfix">
110+
<para>
111+
keep downstream keepalive disabled after a duplicate or invalid upstream
112+
"Content-Length"; a clean upstream close no longer re-enables it, so an
113+
HTTP/1.1 client cannot reuse the connection past the re-framed response. The
114+
complete body still ends with its terminal chunk -- body truncation is now
115+
tracked separately from the inconsistent flag.
116+
</para>
117+
</change>
118+
100119
<change type="change">
101120
<para>
102121
upgrade contrib njs to 1.0.0.

src/nxt_cgroup.c

Lines changed: 79 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -34,12 +34,64 @@ nxt_cgroup_make_procs_path(char *cgprocs, size_t len)
3434
}
3535

3636

37+
/*
38+
* Remove the leaf cgroup directory at cgpath and its now-empty ancestors,
39+
* walking up until cgroot (the parent process's own cgroup) is reached.
40+
* rmdir() fails harmlessly on any directory that is non-empty or was not
41+
* created here. cgpath is truncated in place.
42+
*/
43+
static void
44+
nxt_cgroup_rmdir_up(char *cgpath, const char *cgroot)
45+
{
46+
char *ptr;
47+
size_t root_len, cgpath_len;
48+
49+
/*
50+
* nxt_mk_cgpath(task, "", ...) builds cgroot with a trailing slash
51+
* (".../<main process cgroup>/") because it appends "/%s" with an empty
52+
* relative dir, whereas cgpath has none. Compare against the
53+
* slash-trimmed root length so the walk stops at the parent's own cgroup
54+
* rather than running up to the cgroup mount root -- previously the
55+
* trailing-slash mismatch let strcmp() never match, and only rmdir()
56+
* failing on the non-empty ancestors kept it from removing them.
57+
*/
58+
root_len = strlen(cgroot);
59+
while (root_len > 0 && cgroot[root_len - 1] == '/') {
60+
root_len--;
61+
}
62+
63+
/*
64+
* Stop at the parent's own cgroup -- an exact, length-checked match so a
65+
* sibling prefix ("<cgroot>-x") cannot match -- and never walk at or above
66+
* NXT_CGROUP_ROOT: an absolute isolation "path" resolves under
67+
* NXT_CGROUP_ROOT rather than under cgroot, so cgroot is never reached and
68+
* without this floor the walk would climb toward the cgroup mount root.
69+
*/
70+
cgpath_len = strlen(cgpath);
71+
72+
while (cgpath_len > sizeof(NXT_CGROUP_ROOT) - 1
73+
&& !(cgpath_len == root_len
74+
&& strncmp(cgpath, cgroot, root_len) == 0))
75+
{
76+
rmdir(cgpath);
77+
78+
ptr = strrchr(cgpath, '/');
79+
if (ptr == NULL) {
80+
break;
81+
}
82+
83+
*ptr = '\0';
84+
cgpath_len = ptr - cgpath; /* truncated in place; new length is O(1) */
85+
}
86+
}
87+
88+
3789
nxt_int_t
3890
nxt_cgroup_proc_add(nxt_task_t *task, nxt_process_t *process)
3991
{
4092
int len;
4193
size_t old_len;
42-
char cgprocs[NXT_MAX_PATH_LEN];
94+
char cgprocs[NXT_MAX_PATH_LEN], cgroot[NXT_MAX_PATH_LEN];
4395
FILE *fp;
4496
nxt_int_t ret;
4597

@@ -78,11 +130,33 @@ nxt_cgroup_proc_add(nxt_task_t *task, nxt_process_t *process)
78130
*/
79131
process->isolation.cgroup.resolved_path = nxt_mp_alloc(process->mem_pool,
80132
old_len + 1);
81-
if (nxt_fast_path(process->isolation.cgroup.resolved_path != NULL)) {
82-
nxt_memcpy(process->isolation.cgroup.resolved_path, cgprocs, old_len);
83-
process->isolation.cgroup.resolved_path[old_len] = '\0';
133+
if (nxt_slow_path(process->isolation.cgroup.resolved_path == NULL)) {
134+
/*
135+
* Without the cached path, nxt_cgroup_cleanup() cannot rmdir the
136+
* directory we just created -- the child's /proc/<pid>/cgroup is gone
137+
* by cleanup time. Remove it now, together with any now-empty parents
138+
* nxt_fs_mkdir_p() just created (up to the parent's own cgroup), and
139+
* fail rather than moving the process into a cgroup that could never be
140+
* cleaned up.
141+
*/
142+
if (nxt_fast_path(nxt_mk_cgpath(task, "", cgroot, 0) != NXT_ERROR)) {
143+
nxt_cgroup_rmdir_up(cgprocs, cgroot);
144+
145+
} else {
146+
/*
147+
* The boundary could not be resolved (e.g. /proc read failure
148+
* under the same memory pressure); at least remove the leaf we
149+
* created rather than leaking it.
150+
*/
151+
(void) rmdir(cgprocs);
152+
}
153+
154+
return NXT_ERROR;
84155
}
85156

157+
nxt_memcpy(process->isolation.cgroup.resolved_path, cgprocs, old_len);
158+
process->isolation.cgroup.resolved_path[old_len] = '\0';
159+
86160
ret = nxt_cgroup_make_procs_path(cgprocs, old_len);
87161
if (nxt_slow_path(ret == NXT_ERROR)) {
88162
return NXT_ERROR;
@@ -108,7 +182,6 @@ nxt_cgroup_proc_add(nxt_task_t *task, nxt_process_t *process)
108182
void
109183
nxt_cgroup_cleanup(nxt_task_t *task, const nxt_process_t *process)
110184
{
111-
char *ptr;
112185
char cgroot[NXT_MAX_PATH_LEN], cgpath[NXT_MAX_PATH_LEN];
113186
nxt_int_t ret;
114187

@@ -142,14 +215,7 @@ nxt_cgroup_cleanup(nxt_task_t *task, const nxt_process_t *process)
142215
return;
143216
}
144217

145-
while (*cgpath != '\0' && strcmp(cgroot, cgpath) != 0) {
146-
rmdir(cgpath);
147-
ptr = strrchr(cgpath, '/');
148-
if (ptr == NULL) {
149-
break;
150-
}
151-
*ptr = '\0';
152-
}
218+
nxt_cgroup_rmdir_up(cgpath, cgroot);
153219
}
154220

155221

src/nxt_h1proto.c

Lines changed: 27 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1642,7 +1642,7 @@ nxt_h1p_chunk_create(nxt_task_t *task, nxt_http_request_t *r, nxt_buf_t *out)
16421642
for (b = out; b != NULL; b = b->next) {
16431643

16441644
if (nxt_buf_is_last(b)) {
1645-
if (r->inconsistent) {
1645+
if (r->truncated) {
16461646
/*
16471647
* The response is truncated -- the upstream closed before the
16481648
* body framing completed (premature chunked EOF, or a
@@ -1651,6 +1651,11 @@ nxt_h1p_chunk_create(nxt_task_t *task, nxt_http_request_t *r, nxt_buf_t *out)
16511651
* so the connection closes after this buffer and the client
16521652
* detects the truncation via the missing terminator, rather
16531653
* than seeing a falsely complete response. #72
1654+
*
1655+
* Keyed on "truncated" not "inconsistent": a complete body
1656+
* flagged inconsistent for another reason (e.g. a duplicate
1657+
* upstream Content-Length) must keep its terminal chunk while
1658+
* still disabling keepalive.
16541659
*/
16551660
break;
16561661
}
@@ -3052,16 +3057,28 @@ nxt_h1p_peer_closed(nxt_task_t *task, void *obj, void *data)
30523057
* if its framing never completed: a Content-Length response short of
30533058
* its declared length (remainder != 0), or a chunked response that
30543059
* never reached the terminal 0\r\n\r\n (!chunked_parse.last). Mark it
3055-
* inconsistent so keepalive is disabled and the client connection is
3056-
* closed once the partial body has been relayed. The missing
3057-
* terminator then lets the client detect the truncation instead of it
3058-
* being masked as a clean end of response. Relaying the partial body
3059-
* and closing -- rather than calling error_handler -- avoids racing a
3060-
* connection reset against the already-buffered status line and body
3061-
* (which could otherwise leave the client with an empty response). #72
3060+
* "truncated" so nxt_h1p_chunk_create() drops the terminal chunk and
3061+
* the client detects the cut instead of it being masked as a clean end
3062+
* of response, and "inconsistent" so keepalive is disabled and the
3063+
* client connection is closed once the partial body has been relayed.
3064+
* Relaying the partial body and closing -- rather than calling
3065+
* error_handler -- avoids racing a connection reset against the
3066+
* already-buffered status line and body (which could otherwise leave
3067+
* the client with an empty response). #72
3068+
*
3069+
* Set rather than assign: an earlier stage may already have marked the
3070+
* response inconsistent for an unrelated reason whose body is complete
3071+
* (e.g. an invalid or duplicate upstream Content-Length in
3072+
* nxt_http_proxy_content_length()). A clean close there must keep
3073+
* keepalive disabled without falsely truncating the framing, so only a
3074+
* genuine short body sets "truncated", and neither flag is ever cleared.
30623075
*/
3063-
r->inconsistent = (h1p->remainder != 0)
3064-
|| (h1p->chunked && !h1p->chunked_parse.last);
3076+
if ((h1p->remainder != 0)
3077+
|| (h1p->chunked && !h1p->chunked_parse.last))
3078+
{
3079+
r->truncated = 1;
3080+
r->inconsistent = 1;
3081+
}
30653082

30663083
r->state->ready_handler(task, r, peer);
30673084

src/nxt_http.h

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -210,6 +210,7 @@ struct nxt_http_request_s {
210210
uint8_t logged; /* 1 bit */
211211
uint8_t header_sent; /* 1 bit */
212212
uint8_t inconsistent; /* 1 bit */
213+
uint8_t truncated; /* 1 bit */
213214
uint8_t error; /* 1 bit */
214215
uint8_t websocket_handshake; /* 1 bit */
215216
uint8_t chunked; /* 1 bit */

test/fake_upstream/README.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,7 @@ single `grep` finds every side of a case. E.g. token `chunked_response`:
7676

7777
| Port | Token | Mode (CLI) | Rust handler | pytest |
7878
|------|-------|-----------|--------------|--------|
79+
| 7979 | `dup_cl` | `dup-cl` | `respond_dup_cl` | `test_proxy_dup_cl_keepalive_disabled` (duplicate upstream Content-Length disables downstream keepalive for a keep-alive client; complete body keeps its terminal chunk) — out of the 7983–7999 block below, which is exhausted; 7980–7982 are `fake_otlp`, so 7974–7979 is the free gap |
7980
| 7983 | `dup_cl` | `dup-cl` | `respond_dup_cl` | `test_proxy_dup_cl_raw` (zero raw `Content-Length` occurrences on the wire, complete re-framed chunked body) |
8081
| 7984 | `dup_cl` | `dup-cl` | `respond_dup_cl` | `test_proxy_dup_cl` (duplicate upstream Content-Length → neither forwarded, body re-framed, #113) |
8182
| 7985 | `chunked_ext` | `chunked-ext` | `respond_chunked_edge` | `test_proxy_chunked_ext` (chunk-extensions stripped, body intact) |
@@ -93,6 +94,10 @@ single `grep` finds every side of a case. E.g. token `chunked_response`:
9394
| 7997 | `slow_drip` | `slow-drip` | `respond_slow_drip` | `test_proxy_chunked_response_slow_drip` (#72 case 5) |
9495
| 7998 | `dup_te` | `dup-te` | `respond_dup_te` | `test_proxy_chunked_response_dup_te` (#72 case 6, nginx/unit#1088) |
9596

97+
> **7999 is not available here.** `test/test_proxy.py` and `test/test_proxy_chunked.py`
98+
> already bind `SERVER_PORT = 7999` for their own upstream, so this registry ends at
99+
> 7998. New `fake_upstream` slots go to the 7974–7979 gap (7980–7982 are `fake_otlp`).
100+
96101
A test pins its port as a module constant referencing this table:
97102

98103
```python

test/test_proxy_dup_cl.py

Lines changed: 84 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -14,13 +14,17 @@
1414
bytes and a complete terminal chunk. The client never sees the two ambiguous
1515
lengths, so there is no framing disagreement to exploit.
1616
17-
Note on the inconsistent flag: nxt_http_proxy_content_length also sets
18-
r->inconsistent, but once the EOF-framed body ends with a clean upstream
19-
close, nxt_h1p_peer_closed recomputes the flag from the (already reset)
20-
framing state, so a keep-alive-disabling close is not independently
21-
observable here. Both tests therefore drive `Connection: close` and assert
22-
the defense that matters on the wire: zero conflicting Content-Length headers
23-
reach the client and the relayed body framing is unambiguous.
17+
Note on the inconsistent flag: nxt_http_proxy_content_length sets
18+
r->inconsistent so the downstream keepalive is disabled. This was previously
19+
clobbered -- nxt_h1p_peer_closed reassigned the flag from the (already reset)
20+
framing state on the clean upstream close, so the connection stayed
21+
keep-alive. That is fixed: peer_closed now only *sets* a separate r->truncated
22+
(which drives the terminal-chunk omission) and never clears r->inconsistent,
23+
so a complete-but-ambiguous body disables keepalive while keeping its terminal
24+
chunk. test_proxy_dup_cl_keepalive_disabled below asserts that directly. The
25+
first two tests drive `Connection: close` and assert the primary defense on
26+
the wire: zero conflicting Content-Length headers reach the client and the
27+
relayed body framing is unambiguous.
2428
2529
Driven by the `dup-cl` mode of the Rust mock upstream (test/fake_upstream/):
2630
it sends `Content-Length: 20` then `Content-Length: 6`, followed by a 20-byte
@@ -55,6 +59,7 @@
5559
# Reserved fake_upstream ports for these cases (see test/fake_upstream/README.md).
5660
UPSTREAM_DUP_CL_PORT = 7984
5761
UPSTREAM_DUP_CL_RAW_PORT = 7983
62+
UPSTREAM_DUP_CL_KA_PORT = 7979
5863

5964
FAKE_UPSTREAM_BIN = '/usr/local/bin/fake_upstream'
6065

@@ -197,3 +202,75 @@ def test_proxy_dup_cl_raw(skip_alert):
197202
finally:
198203
proc.terminate()
199204
proc.wait()
205+
206+
207+
@_skipif_no_fake_upstream
208+
def test_proxy_dup_cl_keepalive_disabled(skip_alert):
209+
# Regression for the inconsistent-flag clobber: a duplicate upstream
210+
# Content-Length must disable downstream keepalive even for a keep-alive
211+
# client, AND the re-framed body must keep its terminal chunk (a complete
212+
# body is not falsely truncated). Before the fix, nxt_h1p_peer_closed
213+
# reassigned r->inconsistent to 0 on the clean upstream close, so the
214+
# connection stayed keep-alive; a naive fix that reused that one flag would
215+
# instead drop the terminal chunk. Both properties are asserted here.
216+
skip_alert(r'upstream sent duplicate Content-Length')
217+
218+
proc = _run(UPSTREAM_DUP_CL_KA_PORT, 'dup-cl')
219+
try:
220+
_conf_proxy(UPSTREAM_DUP_CL_KA_PORT)
221+
222+
# Explicit keep-alive request: the default client sends Connection:
223+
# close, which would mask whether the server disables keepalive.
224+
sock = client.get(
225+
port=8080,
226+
headers={'Host': 'localhost', 'Connection': 'keep-alive'},
227+
no_recv=True,
228+
)
229+
sock.settimeout(10)
230+
231+
data = b''
232+
closed = False
233+
try:
234+
while True:
235+
part = sock.recv(4096)
236+
if not part:
237+
closed = True
238+
break
239+
data += part
240+
# Once the full response is in, only a short grace period is
241+
# needed to observe the server-initiated close; shorten the
242+
# timeout so a *failing* run (keepalive left enabled) does not
243+
# block for the full 10s.
244+
if b'0\r\n\r\n' in data:
245+
sock.settimeout(1)
246+
except socket.timeout:
247+
closed = False
248+
finally:
249+
sock.close()
250+
251+
assert data[:12] == b'HTTP/1.1 200', f'status line: {data[:40]!r}'
252+
253+
sep = data.index(b'\r\n\r\n')
254+
head = data[:sep].lower()
255+
body = data[sep + 4:]
256+
257+
# Keepalive is disabled despite the client's keep-alive request: the
258+
# server closes the connection after the response. No explicit
259+
# "Connection: close" header is emitted -- nxt_h1p_request_header_send()
260+
# picks the Connection header from h1p->keepalive before
261+
# nxt_h1p_request_close() applies "keepalive &= !inconsistent" -- so the
262+
# socket close is the observable signal here.
263+
assert closed, 'server must close the connection after a dup-CL response'
264+
265+
# The complete body keeps its terminal chunk: the fix decouples the
266+
# keepalive-disable (r->inconsistent) from truncation (r->truncated),
267+
# so framing stays unambiguous rather than being falsely cut.
268+
assert head.count(b'content-length') == 0, (
269+
f'conflicting Content-Length must not reach the client: {data[:sep]!r}'
270+
)
271+
assert b'transfer-encoding: chunked' in head, f'not re-framed: {head!r}'
272+
assert body.endswith(b'0\r\n\r\n'), f'terminal chunk missing: {body!r}'
273+
assert _dechunk(body) == BODY.encode(), f'relayed body mismatch: {body!r}'
274+
finally:
275+
proc.terminate()
276+
proc.wait()

0 commit comments

Comments
 (0)