Skip to content

Commit 994ad9e

Browse files
crispasr integrationclaude
andcommitted
docs(pcs): the cap dump's trailing bits are don't-care
Mirrors CrispEmbed 75b1cf53 into both copies (test-copies-in-sync requires it). The truecase head always emits 16 bits, but the reconstruction reads bit `c` only for character `c` of the piece — the tail is padding the model fills arbitrarily. A consumer comparing all 16 will fail a CORRECT artifact: the q4_k-imatrix build differs from the f32 reference on bits 4..12 of `▁ok` (3 characters) while producing byte-identical text. CrispEmbed's pcs_parity.py did exactly that until it was truncated to the piece length, so the note is worth carrying next to the dump rather than only in the consumer. Comment-only; verified test-copies-in-sync 47/47 and format.sh OK on 583 files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 36dd37b commit 994ad9e

2 files changed

Lines changed: 14 additions & 4 deletions

File tree

crisp_punc/src/pcs.cpp

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -837,8 +837,13 @@ static PCSResult pcs_run(pcs_context& ctx, const std::vector<int>& token_ids) {
837837
// Two ints per token: thresholded boundary, then the hard argmax.
838838
dump("PCS_DUMP_SEG",
839839
[&](FILE* fp, int t) { fprintf(fp, "%d %d", result.sbd_preds[t] ? 1 : 0, seg_argmax[t] ? 1 : 0); });
840-
// 16 bits per token, MSB-first by character position — the truecase head
841-
// is per-CHARACTER within the token, not one flag per token.
840+
// 16 bits per token, by character position — the truecase head is
841+
// per-CHARACTER within the token, not one flag per token. ⚠ Only the
842+
// first `len(piece)` bits are meaningful; the reconstruction reads bit
843+
// c for character c and never looks further, so the tail is padding the
844+
// model fills arbitrarily and a consumer MUST truncate before comparing.
845+
// Comparing all 16 fails a correct artifact (see CrispEmbed
846+
// tests/pcs_parity.py, which did exactly that until it was fixed).
842847
dump("PCS_DUMP_CAP", [&](FILE* fp, int t) {
843848
for (size_t c = 0; c < result.cap_preds[t].size(); c++) {
844849
fputc(result.cap_preds[t][c] ? '1' : '0', fp);

src/pcs.cpp

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -837,8 +837,13 @@ static PCSResult pcs_run(pcs_context& ctx, const std::vector<int>& token_ids) {
837837
// Two ints per token: thresholded boundary, then the hard argmax.
838838
dump("PCS_DUMP_SEG",
839839
[&](FILE* fp, int t) { fprintf(fp, "%d %d", result.sbd_preds[t] ? 1 : 0, seg_argmax[t] ? 1 : 0); });
840-
// 16 bits per token, MSB-first by character position — the truecase head
841-
// is per-CHARACTER within the token, not one flag per token.
840+
// 16 bits per token, by character position — the truecase head is
841+
// per-CHARACTER within the token, not one flag per token. ⚠ Only the
842+
// first `len(piece)` bits are meaningful; the reconstruction reads bit
843+
// c for character c and never looks further, so the tail is padding the
844+
// model fills arbitrarily and a consumer MUST truncate before comparing.
845+
// Comparing all 16 fails a correct artifact (see CrispEmbed
846+
// tests/pcs_parity.py, which did exactly that until it was fixed).
842847
dump("PCS_DUMP_CAP", [&](FILE* fp, int t) {
843848
for (size_t c = 0; c < result.cap_preds[t].size(); c++) {
844849
fputc(result.cap_preds[t][c] ? '1' : '0', fp);

0 commit comments

Comments
 (0)