feat(pcs): PCS_DUMP_PRE / _SEG / _CAP — compare all four heads, not one

Mirrors CrispEmbed 381857ab into both copies of pcs.cpp here (the shared
crisp_punc one and the src/ fallback), which test-copies-in-sync requires and
verifies.

PCS_DUMP_LOGITS covered post-punc only, so the pre, sentence-boundary and
truecase heads could be checked solely through their effect on the restored
string. Measured against a reference dumped from the model's own driving code
(the punctuators package, via CrispEmbed's tools/dump_pcs_reference.py), plain
q4_k gets all 67 post-punc decisions right and still turns "I'm OK" into
"I'm ok" — visible, but with one head dumped, unattributable. With all four:

  post 67/67   pre 67/67   seg 67/67   cap 66/67 (token 37)

One token, one head: q4_k leaves the punctuation path alone and the whole cost
is the truecase head.

Written in one block after every head has finished, so the dumps are aligned by
construction rather than by three call sites agreeing about ordering. Append
mode matches PCS_DUMP_LOGITS.

PCS_DUMP_SEG deliberately has two columns — the 0.05-thresholded boundary (the
ONNX seg_preds output) and the hard argmax that conditions the truecase head —
because a bug in either produces wrong text and they disagree by design.
PCS_DUMP_CAP is per CHARACTER of the token, ▁ included, not one flag per token.

Verified: test-copies-in-sync 47/47, format.sh OK on 583 files, and CrispEmbed
built against this copy scores 67/67 on every head.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: crispasr integration

crisp_punc/src/pcs.cpp

@@ -36,6 +36,7 @@
 #include <cstdio>
 #include <cstdlib>
 #include <cstring>
+#include <functional>
 #include <map>
 #include <string>
 #include <vector>
@@ -796,6 +797,55 @@ static PCSResult pcs_run(pcs_context& ctx, const std::vector<int>& token_ids) {
         }
     }
 
+    // Diff harness for the three heads PCS_DUMP_LOGITS does not reach.
+    //
+    // That hook covers post-punc only, so the pre / sentence-boundary / truecase
+    // heads could previously be checked ONLY through their effect on the
+    // restored string. That is a real gap and it showed: plain q4_k gets all 67
+    // post-punc decisions right on the reference corpus and still turns
+    // "I'm OK" into "I'm ok" — a truecase-head regression that surfaced as a text
+    // diff, with nothing to say which head produced it.
+    //
+    // Written here, after every head has finished, so the four dumps are aligned
+    // by construction rather than by three call sites agreeing about ordering.
+    // Append mode matches PCS_DUMP_LOGITS — a consumer must delete the file
+    // first, or a stale run silently shifts every comparison.
+    //
+    // ⚠ The seg head yields TWO different quantities and both are dumped,
+    // because a bug in either produces wrong text and they disagree by design:
+    //   seg  = softmax(logits)[boundary] > 0.05, the ONNX `seg_preds` output —
+    //          a low tuned threshold, NOT argmax
+    //   arg  = the hard argmax, which is what conditions the truecase head
+    // Dumping only one of them would make a mismatch in the other invisible.
+    {
+        auto dump = [&](const char* env, const std::function<void(FILE*, int)>& row) {
+            const char* path = std::getenv(env);
+            if (!path)
+                return;
+            FILE* fp = fopen(path, "a");
+            if (!fp)
+                return;
+            for (int t = 0; t < N; t++) {
+                row(fp, t);
+                fputc('\n', fp);
+            }
+            fclose(fp);
+        };
+        // One int per token: the pre-punc argmax (0 = <NULL>, 1 = the inverted
+        // Spanish opener).
+        dump("PCS_DUMP_PRE", [&](FILE* fp, int t) { fprintf(fp, "%d", result.pre_preds[t]); });
+        // Two ints per token: thresholded boundary, then the hard argmax.
+        dump("PCS_DUMP_SEG",
+             [&](FILE* fp, int t) { fprintf(fp, "%d %d", result.sbd_preds[t] ? 1 : 0, seg_argmax[t] ? 1 : 0); });
+        // 16 bits per token, MSB-first by character position — the truecase head
+        // is per-CHARACTER within the token, not one flag per token.
+        dump("PCS_DUMP_CAP", [&](FILE* fp, int t) {
+            for (size_t c = 0; c < result.cap_preds[t].size(); c++) {
+                fputc(result.cap_preds[t][c] ? '1' : '0', fp);
+            }
+        });
+    }
+
     ggml_free(ctx0);
     return result;
 }

src/pcs.cpp

@@ -36,6 +36,7 @@
 #include <cstdio>
 #include <cstdlib>
 #include <cstring>
+#include <functional>
 #include <map>
 #include <string>
 #include <vector>
@@ -796,6 +797,55 @@ static PCSResult pcs_run(pcs_context& ctx, const std::vector<int>& token_ids) {
         }
     }
 
+    // Diff harness for the three heads PCS_DUMP_LOGITS does not reach.
+    //
+    // That hook covers post-punc only, so the pre / sentence-boundary / truecase
+    // heads could previously be checked ONLY through their effect on the
+    // restored string. That is a real gap and it showed: plain q4_k gets all 67
+    // post-punc decisions right on the reference corpus and still turns
+    // "I'm OK" into "I'm ok" — a truecase-head regression that surfaced as a text
+    // diff, with nothing to say which head produced it.
+    //
+    // Written here, after every head has finished, so the four dumps are aligned
+    // by construction rather than by three call sites agreeing about ordering.
+    // Append mode matches PCS_DUMP_LOGITS — a consumer must delete the file
+    // first, or a stale run silently shifts every comparison.
+    //
+    // ⚠ The seg head yields TWO different quantities and both are dumped,
+    // because a bug in either produces wrong text and they disagree by design:
+    //   seg  = softmax(logits)[boundary] > 0.05, the ONNX `seg_preds` output —
+    //          a low tuned threshold, NOT argmax
+    //   arg  = the hard argmax, which is what conditions the truecase head
+    // Dumping only one of them would make a mismatch in the other invisible.
+    {
+        auto dump = [&](const char* env, const std::function<void(FILE*, int)>& row) {
+            const char* path = std::getenv(env);
+            if (!path)
+                return;
+            FILE* fp = fopen(path, "a");
+            if (!fp)
+                return;
+            for (int t = 0; t < N; t++) {
+                row(fp, t);
+                fputc('\n', fp);
+            }
+            fclose(fp);
+        };
+        // One int per token: the pre-punc argmax (0 = <NULL>, 1 = the inverted
+        // Spanish opener).
+        dump("PCS_DUMP_PRE", [&](FILE* fp, int t) { fprintf(fp, "%d", result.pre_preds[t]); });
+        // Two ints per token: thresholded boundary, then the hard argmax.
+        dump("PCS_DUMP_SEG",
+             [&](FILE* fp, int t) { fprintf(fp, "%d %d", result.sbd_preds[t] ? 1 : 0, seg_argmax[t] ? 1 : 0); });
+        // 16 bits per token, MSB-first by character position — the truecase head
+        // is per-CHARACTER within the token, not one flag per token.
+        dump("PCS_DUMP_CAP", [&](FILE* fp, int t) {
+            for (size_t c = 0; c < result.cap_preds[t].size(); c++) {
+                fputc(result.cap_preds[t][c] ? '1' : '0', fp);
+            }
+        });
+    }
+
     ggml_free(ctx0);
     return result;
 }