Skip to content

fix(kickstart): decode UTF-8 by hand instead of locale-aware fgetwc() - #2255

Open
mayani wants to merge 1 commit into
mainfrom
bug/2250-kickstart-truncates-yaml-on-non-ascii
Open

mayani wants to merge 1 commit into
mainfrom
bug/2250-kickstart-truncates-yaml-on-non-ascii

Conversation

@mayani

@mayani mayani commented Sep 16, 2026

Copy link
Copy Markdown
Member

Summary

Under a C/POSIX locale, fgetwc() treats any byte >= 0x80 as an invalid multibyte
sequence and returns WEOF, indistinguishable from real EOF. yamldump() and
printYAMLStatInfo()'s character-counting loop both looped on fgetwc(), so a
single non-ASCII byte anywhere in captured stdout/stderr silently truncated the
rest of the YAML report — while data_truncated was computed purely from byte
size and never reflected it.

  • Added yamlgetutf8() / yamlpututf8() in utils.c to decode/encode UTF-8 by
    hand from raw bytes, independent of locale.
  • Undecodable bytes (including UTF-8-encoded surrogates, which are never valid)
    are now skipped instead of stopping the dump, and data_truncated is derived
    from a first pass that also tracks whether any bytes had to be skipped this way.
  • yamldump() also now re-indents on YAML 1.1's other line-break characters
    (NEL, LS, PS), which previously could pass through raw and de-indent content
    out of the literal block once locale no longer masked the issue (found during
    adversarial review — a job could otherwise inject sibling YAML keys into its
    own invocation record via a crafted stdout/stderr byte sequence).

Verified with manual repro (valid UTF-8 em-dash case from the issue's repro,
invalid-byte case, CESU-8/WTF-8 surrogate case, and a YAML block-scalar
injection attempt parsed back with PyYAML) plus the full pegasus-kickstart
C test suite.

Skipped review findings

Autonomous review (simplify, security, general-correctness, Codex review,
Codex adversarial-review) surfaced a few additional items that are pre-existing
behavior unrelated to this locale bug, judged out of scope for this fix:

  • statinfo.c:513-533 — the two-pass fd counting/skip loop is duplicated and
    now slightly costlier per call than the old fgetwc version; predates this
    change (same double-pass shape existed before), a single-pass rewrite is a
    separate cleanup.
  • utils.c:88 yamlgetutf8 — its 3-state return (0/1/2) could be collapsed to
    a bool, but that would lose the "skip invalid, keep reading" signal this fix
    relies on; kept as-is.
  • statinfo.c:518,536 — two FILE* opened on the same duped fd via fdopen,
    the first never fclose()'d; pre-existing pattern (existed with fgetwc
    too), not introduced by this diff.
  • statinfo.c:493-524-B/data_section_size is documented as a byte limit
    but the counting loop counts decoded UTF-8 code points, so multi-byte output
    can exceed the configured limit by up to ~4x. This predates pegasus-kickstart silently truncates YAML report on non-ASCII bytes under C locale #2250 (fgetwc
    counted wide chars identically) — making the limit byte-accurate is a larger
    change than this locale fix warrants; worth a follow-up issue.

e2e note

e2e (pipeline 7116) had a single failure unrelated to this change: Performance Tests: [014-planner-performance-100k, launch-bamboo-test-planner-only] — a
Java planner timing test (100k-job workflow planning took 1760s against an
upper limit of 1600s), which doesn't touch pegasus-kickstart at all. Confirmed
via trace as CI-runner timing flakiness, not a regression from this diff.

Test plan

  • make build-c (pegasus-kickstart, pegasus-cluster, pegasus-keg)
  • packages/pegasus-kickstart/test/test.sh — 37/37 applicable tests pass
    (the one skipped test requires the Go pegasus-integrity binary, not
    built by make build-c in this environment; unrelated to this change)
  • Manual repro of the original issue's em-dash case, an invalid-byte case,
    a CESU-8 surrogate case, and a YAML injection attempt — all verified
    fixed
  • e2e (GitLab pipeline 7116) — green except for the unrelated planner
    timing flake noted above

Closes #2250

Under a C/POSIX locale, fgetwc() treats any byte >= 0x80 as an invalid
multibyte sequence and returns WEOF, indistinguishable from real EOF.
yamldump() and printYAMLStatInfo()'s character-counting loop looped on
fgetwc(), so a single non-ASCII byte in captured stdout/stderr silently
truncated the rest of the YAML report, while data_truncated was computed
purely from byte size and never reflected it.

Add yamlgetutf8()/yamlpututf8() to decode/encode UTF-8 by hand from raw
bytes, independent of locale. Undecodable bytes (including UTF-8-encoded
surrogates, which are never valid) are now skipped instead of stopping
the dump, and data_truncated is derived from a first pass that also
tracks whether any bytes had to be skipped this way. yamldump() also now
re-indents on YAML 1.1's other line-break characters (NEL, LS, PS), which
previously could pass through raw and de-indent content out of the
literal block once locale no longer masked the issue.

Refs #2250
@mayani mayani self-assigned this Sep 16, 2026
@mayani
mayani requested a review from rynge September 16, 2026 05:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pegasus-kickstart silently truncates YAML report on non-ASCII bytes under C locale

1 participant