Skip to content

Commit 6ba825d

Browse files
committed
feat(reads): :claims op — daemon-first CLI reads (thread 019f2190)
Kill the per-process cold fold: read verbs fetch the coordinator's already-folded live view instead of re-folding the whole log per CLI invocation (~700ms of every read on the 11k-line tern log). - cnf_coord_daemon: new :claims op — whole live view as [l p r] wire triples IN FOLD EMISSION ORDER (contract), cached per version, served after maybe-reload! so it is exactly as fresh as the flat log; :log echo lets a client refuse a mismatched daemon. - fold: refold-order — re-key a set-equal live view into fold's hash-map emission order (bb and the JVM share PersistentHashMap, so daemon order == CLI order; byte-identical downstream output). - rt: coord-live-claims client — asks {:fmt :json} (bb decodes the ~2MB payload 12x faster than EDN); returns [] on daemon-down / unknown-op / log-mismatch, the whole capability handshake (callers fall back cold). - kernel: single? via map membership (ran once per assertion in every fold/refold — the vec scan was a measurable slice of the read path). - main: cmd-show reads through live-claims; prefix resolution over the index (O(subjects x claims) flat scan was ~2s -> ~40ms), byte-identical match order. Measured (10.8k-claim live-size copy, isolated daemon :7999): fram show 744ms -> 160ms full-id, 2.2s -> 208ms prefix; all read verbs byte- identical warm vs cold; out-of-band appends visible warm; clean cold fallback with daemon down / old daemon / wrong log. Drop-in gate: torn- tail freshness check fails identically on unmodified main (pre-existing).
1 parent 421c1e5 commit 6ba825d

9 files changed

Lines changed: 145 additions & 10 deletions

File tree

cnf_coord_daemon.clj

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@
3333
(def co (atom nil)) ; {:store :log :lock} — reified canonical (v2 log)
3434
(def flat-log (atom nil)) ; the flat log, now a PROJECTION the cold CLI folds
3535
(def cache (atom {:index nil :version -1}))
36+
(def claims-wire-cache (atom {:version -1 :triples nil})) ; :claims op — fold-ordered [l p r] wire triples, per version
3637
(def subscribers (atom []))
3738
(def dlock (Object.)) ; serializes reload + writes + reads (drop-in mode)
3839
(def flat-mtime (atom nil)) ; last-seen flat-log stamp (to detect external edits)
@@ -1257,6 +1258,28 @@
12571258
:inc-triples (count (:triples (:idx inc))) :fresh-triples (count (:triples fidx))
12581259
:version (current-seq @co)})
12591260
:status {:version (current-seq @co) :claims (count (c/current-claims (:store @co))) :log (or @flat-log (:log @co))}
1261+
;; :claims — the WHOLE live view as flat [l p r] triples IN FOLD EMISSION ORDER
1262+
;; (fram.fold/refold-order), for daemon-first CLI reads (thread 019f2190): the
1263+
;; client feeds these straight into its kernel index instead of paying the
1264+
;; per-process cold fold. Ordering here is part of the op's CONTRACT — the
1265+
;; client renders byte-identical output without re-ordering, and bb shares
1266+
;; clojure.lang.PersistentHashMap with the JVM so the hash order agrees.
1267+
;; Computed AFTER maybe-reload! above (exactly as fresh as the flat log at
1268+
;; request time) and cached per version — a read storm re-serializes, never
1269+
;; re-orders. :log echoes which log this daemon serves so a client can refuse
1270+
;; a mismatched daemon (fram.rt/coord-live-claims checks it). Clients ask with
1271+
;; {:fmt :json} — bb decodes the ~2MB payload ~12x faster as JSON than as EDN.
1272+
:claims (let [v (current-seq @co)
1273+
cc @claims-wire-cache
1274+
triples (if (= v (:version cc))
1275+
(:triples cc)
1276+
(let [ts (mapv (fn [c] [(:l c) (:p c) (:r c)])
1277+
(fold/refold-order (reified->claims @co)))]
1278+
(reset! claims-wire-cache {:version v :triples ts})
1279+
ts))]
1280+
{:version v
1281+
:log (or @flat-log (:log @co))
1282+
:claims triples})
12601283
;; thread 019f100f-7fff — snapshot/compaction surface:
12611284
;; :snapshot writes a checkpoint (dump-log! image + @snapshot:<seq> claims);
12621285
;; :snapshot-reconcile is the gate (live store == from-scratch whole migrate).

out/fram/fold.clj

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,3 +60,6 @@
6060

6161
(defn fold-latest [asserts]
6262
(filterv (fn [v] (= (:op v) "assert")) (vec (vals (keyed-latest asserts)))))
63+
64+
(defn refold-order [claims]
65+
(vec (vals (reduce (fn [m c] (assoc m (key-of (->Assertion 0 "assert" (:l c) (:p c) (:r c) "live")) c)) {} claims))))

out/fram/kernel.clj

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,8 +24,10 @@
2424
(defn ^String vocab-fingerprint []
2525
(str "single=" (sorted-join single-valued) " |terminal=" (sorted-join terminal-preds) " |withdrawn=" (sorted-join withdrawn-preds)))
2626

27+
(def single-valued-set (reduce (fn [m p] (assoc m p true)) {} single-valued))
28+
2729
(defn ^Boolean single? [^String p]
28-
(or (vec-contains? single-valued p) (and (string? p) (str/starts-with? p "emoji_"))))
30+
(or (some? (get single-valued-set p)) (and (string? p) (str/starts-with? p "emoji_"))))
2931

3032
(defrecord Claim [l p r])
3133

out/fram/main.clj

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,10 @@
1212
(defn- ^String short-id [^String te]
1313
(if (str/starts-with? te "@") (subs te 1) te))
1414

15+
(defn- live-claims [^String log]
16+
(let [warm (fram.rt/coord-live-claims (fram.rt/coord-port) log)]
17+
(if (empty? warm) (:claims (fold/fold (fram.rt/read-log log))) warm)))
18+
1519
(defn- ^String claim-sig [c]
1620
(str (:l c) "|" (:p c) "|" (:r c)))
1721

@@ -56,11 +60,10 @@
5660
(println (str "\n" (count problems) " violation(s)."))))))
5761

5862
(defn cmd-show [^String log ^String id]
59-
(let [f (fold/fold (fram.rt/read-log log))
60-
claims (:claims f)
63+
(let [claims (live-claims log)
6164
te (str "@" id)
6265
exact (k/q-by-l claims te)
63-
matches (if (or (not (empty? exact)) (str/blank? id)) [] (filterv (fn [t] (str/starts-with? (short-id t) id)) (k/thread-ids claims)))]
66+
matches (if (or (not (empty? exact)) (str/blank? id)) [] (filterv (fn [t] (str/starts-with? (short-id t) id)) (k/thread-ids-i (k/build-index claims))))]
6467
(cond
6568
(not (empty? exact)) (doseq [c exact]
6669
(println (str " " (:p c) " " (:r c))))

out/fram/rt.clj

Lines changed: 31 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,8 @@
1010
[clojure.edn :as edn]
1111
[clojure.java.io :as io]
1212
[cheshire.core :as cheshire]
13-
[fram.fold :as fold]))
13+
[fram.fold :as fold]
14+
[fram.kernel :as kernel]))
1415

1516
;; Serialize any value (records serialize as objects keyed by field name; vectors
1617
;; as arrays) to JSON — the engine's structured-output path for the MCP edge.
@@ -310,6 +311,35 @@
310311
(defn coord-callers [port te] (warm-read port {:op :callers :te te})) ; -> {:callers [...]} | nil
311312
(defn coord-resolved [port te pred] (warm-read port {:op :resolved :te te :p pred})) ; -> {:value :members :ambiguous? :values} | nil — surfaces multiplicity (#3)
312313

314+
;; :claims — the daemon's WHOLE live view as [l p r] triples: the daemon-first read
315+
;; path (thread 019f2190). The CLI rebuilds its kernel index from this instead of
316+
;; paying the per-process cold fold (read-log EDN parse + fold ≈ 700ms on the 11k-line
317+
;; tern log). The daemon serves the triples IN FOLD EMISSION ORDER (its contract —
318+
;; fram.fold/refold-order, cached per version), so the records returned here feed
319+
;; build-index directly and every listing stays byte-identical to the cold fold's.
320+
;; Asked with {:fmt :json} DELIBERATELY: bb parses the ~2MB payload ~12x faster as
321+
;; JSON (cheshire, native) than as EDN (measured 15ms vs 186ms).
322+
;; Returns a (Vec kernel/Claim), or [] when the warm path is unavailable — daemon
323+
;; down, an older daemon without the op (its {"error" ...} reply has no "claims"),
324+
;; or a daemon serving a DIFFERENT log than the caller's (the :log echo mismatches —
325+
;; never silently read another store). [] is safe as the sentinel: a real log always
326+
;; folds to a non-empty view (an empty log has no daemon serving it worth trusting).
327+
;; Callers fall back to the cold fold on [].
328+
(defn coord-live-claims [port log]
329+
(try
330+
(with-open [s (coord-socket (connect-host) port)]
331+
(let [w (io/writer (.getOutputStream s))
332+
r (io/reader (.getInputStream s))]
333+
(.write w (str (pr-str {:op :claims :fmt :json}) "\n"))
334+
(.flush w)
335+
(let [resp (cheshire/parse-string (.readLine r))]
336+
(if (and (map? resp)
337+
(= log (get resp "log"))
338+
(vector? (get resp "claims")))
339+
(mapv (fn [t] (kernel/->Claim (nth t 0) (nth t 1) (nth t 2))) (get resp "claims"))
340+
[]))))
341+
(catch Exception _ [])))
342+
313343
;; subscribe + stream commit events (one EDN line each) until disconnect
314344
(defn coord-watch [port]
315345
(with-open [s (coord-socket (connect-host) port)] ; honors FRAM_CONNECT + mTLS like coord-rt

src/fram/fold.bclj

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,3 +76,22 @@
7676
(defn fold-latest [asserts :- (Vec Assertion)] :- (Vec Latest)
7777
(filterv (fn [v :- Latest] :- Bool (= (:op v) "assert"))
7878
(vec (vals (keyed-latest asserts)))))
79+
80+
;; Reconstruct fold's emission ORDER from a set-equal live view. The coordinator's
81+
;; :claims op serves its warm store through this (cached per version), so the CLI's
82+
;; daemon-first read path renders byte-identically to the cold fold. fold emits
83+
;; (vals keyed-latest) — hash-map iteration order over the key-of strings — and
84+
;; downstream output order (build-index subjects, every projection listing)
85+
;; inherits it. Re-keying a set-equal claim vec through the SAME key-of yields the
86+
;; same relative order: an entry's position in a persistent hash map is a function
87+
;; of its own key hash, not of insertion order; the cold map's extra retract-latest
88+
;; keys (skipped when fold emits claims) don't reorder the surviving entries; and
89+
;; bb + the JVM share clojure.lang.PersistentHashMap, so daemon-computed order ==
90+
;; CLI-computed order. (The synthetic Assertion exists only to reuse key-of
91+
;; verbatim — same \001 separator bytes.)
92+
(defn refold-order [claims :- (Vec k/Claim)] :- (Vec k/Claim)
93+
(vec (vals (reduce
94+
(fn [m :- (Map String k/Claim) c :- k/Claim] :- (Map String k/Claim)
95+
(assoc m (key-of (->Assertion 0 "assert" (:l c) (:p c) (:r c) "live")) c))
96+
{}
97+
claims))))

src/fram/kernel.bclj

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -89,13 +89,21 @@
8989
" |terminal=" (sorted-join terminal-preds)
9090
" |withdrawn=" (sorted-join withdrawn-preds)))
9191

92+
;; Map-backed membership for single?: it runs once per assertion in every
93+
;; fold/refold (and per claim in the daemon's migrate/tail-fold), so the
94+
;; vec-contains? linear scan over the ~23-pred vocab was a measurable slice of
95+
;; the whole read path (~40ms of a 10.7k-claim refold on bb).
96+
(def single-valued-set :- (Map String Bool)
97+
(reduce (fn [m :- (Map String Bool) p :- String] :- (Map String Bool) (assoc m p true))
98+
{} single-valued))
99+
92100
;; `emoji_*` are presentation-config predicates (e.g. @ui emoji_blocked "🧱") —
93101
;; single-valued by prefix so re-configuring replaces, with no per-key listing.
94102
(defn single? [p :- String] :- Bool
95103
;; (string? p) guards str/starts-with? against a non-string p — defense in depth
96104
;; so a stray/torn record can't NPE the whole cold-read path (fold already drops
97105
;; torn lines; this is the belt to that suspenders).
98-
(or (vec-contains? single-valued p) (and (string? p) (str/starts-with? p "emoji_"))))
106+
(or (some? (get single-valued-set p)) (and (string? p) (str/starts-with? p "emoji_"))))
99107

100108
;; --- the claim -------------------------------------------------------------
101109

src/fram/main.bclj

Lines changed: 20 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727
(declare-extern fram.rt/coord-retract [Int String String String Int -> String])
2828
(declare-extern fram.rt/coord-port [-> Int])
2929
(declare-extern fram.rt/coord-status [Int -> String])
30+
(declare-extern fram.rt/coord-live-claims [Int String -> (Vec k/Claim)])
3031
(declare-extern fram.rt/coord-watch [Int -> Nil])
3132
(declare-extern fram.rt/history [String String -> Nil])
3233
(declare-extern fram.rt/parse-edn [String -> Any])
@@ -36,6 +37,18 @@
3637
(defn- short-id [te :- String] :- String
3738
(if (str/starts-with? te "@") (subs te 1) te))
3839

40+
;; --- daemon-first read (thread 019f2190) -------------------------------------
41+
;; The current claim set: the coordinator's warm live view when a daemon serves
42+
;; THIS log (served in fold emission order — the :claims op contract — so
43+
;; downstream output is byte-identical to the cold fold's), else the cold fold.
44+
;; [] from coord-live-claims is the whole capability handshake — daemon down /
45+
;; old daemon / different log.
46+
(defn- live-claims [log :- String] :- (Vec k/Claim)
47+
(let [warm (fram.rt/coord-live-claims (fram.rt/coord-port) log)]
48+
(if (empty? warm)
49+
(:claims (fold/fold (fram.rt/read-log log)))
50+
warm)))
51+
3952
;; --- claim-set signatures (divergence detection for the import/export guards) -
4053
(defn- claim-sig [c :- k/Claim] :- String (str (:l c) "|" (:p c) "|" (:r c)))
4154
(defn- sig-set [claims :- (Vec k/Claim)] :- (Vec String) (vec (sort (mapv claim-sig claims))))
@@ -114,14 +127,18 @@
114127
;; (it would match every thread), and an ambiguous prefix lists the candidates and
115128
;; picks none.
116129
(defn cmd-show [log :- String id :- String] :- Nil
117-
(let [f (fold/fold (fram.rt/read-log log))
118-
claims (:claims f)
130+
(let [claims (live-claims log)
119131
te (str "@" id)
120132
exact (k/q-by-l claims te)
121133
matches (if (or (not (empty? exact)) (str/blank? id))
122134
[]
135+
;; prefix resolution over the INDEX: (k/thread-ids claims) is a
136+
;; flat one-per-subject scan — O(subjects × claims), ~2s on the
137+
;; 10.7k-claim log — while thread-ids-i over build-index is
138+
;; O(claims). Same subjects, same uniq claim order, so the match
139+
;; list (and the ambiguous-prefix listing) is byte-identical.
123140
(filterv (fn [t :- String] :- Bool (str/starts-with? (short-id t) id))
124-
(k/thread-ids claims)))]
141+
(k/thread-ids-i (k/build-index claims))))]
125142
(cond
126143
[(not (empty? exact))
127144
(doseq [c exact] (println (str " " (:p c) " " (:r c))))]

src/fram/rt.clj

Lines changed: 31 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,8 @@
1010
[clojure.edn :as edn]
1111
[clojure.java.io :as io]
1212
[cheshire.core :as cheshire]
13-
[fram.fold :as fold]))
13+
[fram.fold :as fold]
14+
[fram.kernel :as kernel]))
1415

1516
;; Serialize any value (records serialize as objects keyed by field name; vectors
1617
;; as arrays) to JSON — the engine's structured-output path for the MCP edge.
@@ -310,6 +311,35 @@
310311
(defn coord-callers [port te] (warm-read port {:op :callers :te te})) ; -> {:callers [...]} | nil
311312
(defn coord-resolved [port te pred] (warm-read port {:op :resolved :te te :p pred})) ; -> {:value :members :ambiguous? :values} | nil — surfaces multiplicity (#3)
312313

314+
;; :claims — the daemon's WHOLE live view as [l p r] triples: the daemon-first read
315+
;; path (thread 019f2190). The CLI rebuilds its kernel index from this instead of
316+
;; paying the per-process cold fold (read-log EDN parse + fold ≈ 700ms on the 11k-line
317+
;; tern log). The daemon serves the triples IN FOLD EMISSION ORDER (its contract —
318+
;; fram.fold/refold-order, cached per version), so the records returned here feed
319+
;; build-index directly and every listing stays byte-identical to the cold fold's.
320+
;; Asked with {:fmt :json} DELIBERATELY: bb parses the ~2MB payload ~12x faster as
321+
;; JSON (cheshire, native) than as EDN (measured 15ms vs 186ms).
322+
;; Returns a (Vec kernel/Claim), or [] when the warm path is unavailable — daemon
323+
;; down, an older daemon without the op (its {"error" ...} reply has no "claims"),
324+
;; or a daemon serving a DIFFERENT log than the caller's (the :log echo mismatches —
325+
;; never silently read another store). [] is safe as the sentinel: a real log always
326+
;; folds to a non-empty view (an empty log has no daemon serving it worth trusting).
327+
;; Callers fall back to the cold fold on [].
328+
(defn coord-live-claims [port log]
329+
(try
330+
(with-open [s (coord-socket (connect-host) port)]
331+
(let [w (io/writer (.getOutputStream s))
332+
r (io/reader (.getInputStream s))]
333+
(.write w (str (pr-str {:op :claims :fmt :json}) "\n"))
334+
(.flush w)
335+
(let [resp (cheshire/parse-string (.readLine r))]
336+
(if (and (map? resp)
337+
(= log (get resp "log"))
338+
(vector? (get resp "claims")))
339+
(mapv (fn [t] (kernel/->Claim (nth t 0) (nth t 1) (nth t 2))) (get resp "claims"))
340+
[]))))
341+
(catch Exception _ [])))
342+
313343
;; subscribe + stream commit events (one EDN line each) until disconnect
314344
(defn coord-watch [port]
315345
(with-open [s (coord-socket (connect-host) port)] ; honors FRAM_CONNECT + mTLS like coord-rt

0 commit comments

Comments
 (0)