[Bug] read_file misdetects valid UTF-8 as binary when CJK text straddles the 1000-byte sample boundary

[hihenen]

Summary

read_file misdetects valid UTF-8 text files as binary when the file contains CJK characters (Korean/Chinese/Japanese) near the 1000-byte mark. The file becomes unreadable to the agent, which then works around it with a python3 <<'PY' heredoc — and that trips the approval rule, so every such read raises a user approval prompt.

Root cause

tools/file_operations.py samples the file with a byte-based head -c 1000 and then treats U+FFFD in the sample as a binary signal:

# L1183 — sampling
sample_cmd = f"head -c 1000 {self._escape_shell_arg(path)} 2>/dev/null"

# L911 — detection
if "�" in content_sample[:1000]:
    return True

head -c cuts on a byte boundary. If a multi-byte character straddles byte 1000, the truncated fragment is decoded with errors="replace" and becomes U+FFFD. The file is then reported as binary even though it is perfectly valid UTF-8.

The comment above that line assumes U+FFFD can only come from genuinely undecodable bytes:

Legitimate UTF-8 text effectively never contains U+FFFD.

That holds for the file, but not for a byte-truncated sample of it.

Reproduction

Verified on v0.20.0. A valid UTF-8 file, 999 bytes of ASCII followed by CJK text:

import subprocess
data = (b'// ' + b'x' * 996) + '한글주석'.encode()   # any CJK text works
open('/tmp/repro.rs', 'wb').write(data)

s = subprocess.run(['head', '-c', '1000', '/tmp/repro.rs'],
                   capture_output=True).stdout
print(repr(s[-6:]))                    # b'xxxxx\xed'  <- truncated mid-character
dec = s.decode('utf-8', errors='replace')
print('�' in dec[:1000])          # True  -> classified as binary

read_file on this path returns is_binary: true, total_lines: 0, file_size: 1011.

This is probabilistic: it depends on where the CJK characters fall relative to byte 1000, so the same file may read fine and then fail after an edit shifts the offset. That makes it look intermittent.

Impact

When read_file refuses, the agent falls back to python3 <<'PY' heredoc, which matches the heredoc rule in tools/approval.py ((python[23]?|perl|ruby|node)\s+<<) and requires interactive approval every time.

Measured on one self-hosted deployment (Slack-facing agent, ~90 days of retained session history):

	count
read_file calls	1,420
classified is_binary: true	27
of those, total_lines: 0 (content unavailable)	27 (all)
heredoc approval prompts raised	32

In one investigation session the agent spent ~5.5 minutes waiting on approval prompts caused by this, and ended by exhausting its iteration budget without producing an answer.

Suggested fix

Truncation artifacts always land at the end of the sample. Stripping them before the check removes the false positives while leaving genuine binary detection intact:

- if "�" in content_sample[:1000]:
+ if "�" in content_sample[:1000].rstrip("�"):
      return True

Verified with the same build:

input	before	after
999B ASCII + CJK (valid UTF-8)	True	False	false positive removed
/bin/ls (real binary)	True	True	detection unchanged
synthetic text with U+FFFD scattered mid-sample	True	True	detection unchanged
ASCII source file	False	False	unchanged

A more thorough alternative is to trim the sample to a valid UTF-8 boundary before decoding (drop a trailing incomplete multi-byte sequence). The rstrip form is the minimal change and covers the observed cases.

Happy to open a PR if this direction looks right.

Environment

• hermes-agent v0.20.0, Linux container
• Affects any repository with CJK comments/strings in the first ~1KB of a file