Skip to content

Commit d15da81

Browse files
Conor-FOSSAclaude
andauthored
[ANE-2795] Fix UTF-8 encoding for ficus output on Windows (#1646)
* Fix UTF-8 encoding crash when reading ficus stderr on Windows Use Conduit with decodeUtf8Lenient for stderr handling, matching the approach already used for stdout. This prevents crashes on Windows where the default system encoding (CP1252) cannot decode UTF-8 characters like box-drawing characters (U+2501) used in ficus progress output. The decodeUtf8Lenient function safely handles any byte sequence by replacing invalid UTF-8 with the Unicode replacement character (U+FFFD) instead of throwing an exception. Fixes: ANE-2795 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix UTF-8 encoding when writing ficus debug logs on Windows Set explicit UTF-8 encoding on debug log file handles. On Windows, file handles default to the system code page (CP1252), which cannot encode Unicode characters like box-drawing characters (U+2501) used in ficus's Rust SPANTRACE output. Without this fix, running `fossa analyze --x-vendetta --debug` on Windows crashes with: commitBuffer: invalid argument (cannot encode character '\9473') This complements the previous fix for reading ficus output by ensuring the writing side also handles UTF-8 correctly. Fixes: ANE-2795 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Simplify stderr accumulator and fix formatting - Remove redundant count tracking from stderr fold (take 50 on a bounded list is already O(1)) - Use Text throughout msg construction instead of String intermediates - Remove trailing blank line to satisfy formatter 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 47bf7c4 commit d15da81

2 files changed

Lines changed: 28 additions & 25 deletions

File tree

Changelog.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
# FOSSA CLI Changelog
22

3+
## Unreleased
4+
5+
- Windows: Fix UTF-8 encoding crash when running `fossa analyze --x-vendetta --debug` on native Windows. The CLI no longer crashes when subprocess output contains Unicode characters. ([#1646](https://github.com/fossas/fossa-cli/pull/1646))
6+
37
## 3.16.5
48

59
- PNPM: Fix pnpm v9 lockfile transitive devDependency classification. Dependencies of devDependencies were incorrectly reported as production dependencies in pnpm v9 projects. ([#1668](https://github.com/fossas/fossa-cli/pull/1668))

src/App/Fossa/Ficus/Analyze.hs

Lines changed: 24 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,7 @@ import Path (Abs, Dir, Path, toFilePath)
5858
import Prettyprinter (pretty)
5959
import Srclib.Types (Locator (..), SourceUnit (..), SourceUnitBuild (..), SourceUnitDependency (..), renderLocator, textToOriginPath)
6060
import System.FilePath ((</>))
61-
import System.IO (Handle, IOMode (WriteMode), hClose, hGetLine, hIsEOF, hPutStrLn, openFile, stderr)
61+
import System.IO (Handle, IOMode (WriteMode), hClose, hPutStrLn, hSetEncoding, openFile, stderr, utf8)
6262
import System.Process.Typed (
6363
createPipe,
6464
getStderr,
@@ -265,7 +265,9 @@ runFicus maybeDebugDir ficusConfig = do
265265
let stdoutPath = debugDir </> "fossa.ficus-stdout.log"
266266
let stderrPath = debugDir </> "fossa.ficus-stderr.log"
267267
stdoutH <- openFile stdoutPath WriteMode
268+
hSetEncoding stdoutH utf8
268269
stderrH <- openFile stderrPath WriteMode
270+
hSetEncoding stderrH utf8
269271
pure (Just stdoutH, Just stderrH)
270272
Nothing ->
271273
-- No debug mode, don't tee to files
@@ -354,32 +356,29 @@ runFicus maybeDebugDir ficusConfig = do
354356
, vendoredDependencyScanResults = vendoredResults
355357
}
356358

359+
-- Use Conduit with decodeUtf8Lenient to safely handle UTF-8 output from ficus.
360+
-- This matches the approach used for stdout and prevents crashes on Windows
361+
-- where the default system encoding (CP1252) cannot decode UTF-8 characters
362+
-- like box-drawing characters (U+2501) used in ficus progress output.
357363
consumeStderr :: Handle -> Maybe Handle -> IO [Text]
358364
consumeStderr handle maybeFile = do
359-
let loop acc (count :: Int) = do
360-
eof <- hIsEOF handle
361-
if eof
362-
then pure (reverse acc) -- Reverse at the end to get correct order
363-
else do
364-
line <- hGetLine handle
365-
-- Tee raw line to file if debug mode
366-
traverse_ (`hPutStrLn` line) maybeFile
367-
now <- getCurrentTime
368-
let timestamp = formatTime defaultTimeLocale "%H:%M:%S.%3q" now
369-
let msg = "[" ++ timestamp ++ "] STDERR " <> line
370-
-- Keep at most the last 50 lines of stderr
371-
-- I came up with 50 lines by looking at a few different error traces and making
372-
-- sure that we captured all of the relevant error output, and then going a bit higher
373-
-- to make sure that we didn't miss anything. I'd rather capture a bit too much than not enough.
374-
-- Use cons (:) for O(1) prepending, track count explicitly for O(1) truncation
375-
let newAcc =
376-
if count >= 50
377-
then take 50 (toText msg : acc)
378-
else toText msg : acc
379-
let newCount = min (count + 1) 50
380-
loop newAcc newCount
381-
loop [] 0
382-
365+
acc <-
366+
Conduit.runConduit $
367+
CC.sourceHandle handle
368+
.| CC.decodeUtf8Lenient
369+
.| CC.linesUnbounded
370+
.| CC.foldM
371+
( \acc line -> do
372+
-- Tee raw line to file if debug mode
373+
traverse_ (\fileH -> hPutStrLn fileH (toString line)) maybeFile
374+
now <- getCurrentTime
375+
-- Keep at most the last 50 lines of stderr (newest first during accumulation)
376+
let ts = toText $ formatTime defaultTimeLocale "%H:%M:%S.%3q" now
377+
let msg = "[" <> ts <> "] STDERR " <> line
378+
pure (take 50 (msg : acc))
379+
)
380+
[]
381+
pure (reverse acc)
383382
displayFicusDebug :: FicusDebug -> Text
384383
displayFicusDebug (FicusDebug FicusMessageData{..}) = ficusMessageDataStrategy <> ": " <> ficusMessageDataPayload
385384
displayFicusError :: FicusError -> Text

0 commit comments

Comments
 (0)