Path-only / by-reference source storage ('the notion of the PDF') — don't re-ingest bytes

[markmhendrickson]

A non-developer evaluator on a 256GB Mac Mini hit source-blob storage anxiety and asked explicitly for a path-only ingestion mode: register a source by its file path WITHOUT copying its bytes into Neotoma (his term: "the notion of the PDF"). Current behavior (confirmed in MCP_SPEC: file_path reads + ingests bytes; FILE_NOT_FOUND/FILE_READ_ERROR) always re-ingests. Mark committed to investigate.

Note: v0.17.0 discard-by-default intake (overflow sink, collapse_by sightings) addresses firehose-pollution but is NOT this ask — that discards; this needs a durable by-reference source row. Distinct from the disk-to-entity write-back feature (bidirectional mirror sync). His strongest open source-storage pain.

Design

Goal: a durable, queryable source row that records WHERE the bytes are, not the bytes — so a large/local file is first-class in the graph without inflating the DB.

API. store (and parse_file) gain source_storage: "inline" | "reference" (default "inline" = today's behavior; fully backward-compatible). With "reference", the server reads the file ONCE to compute content_hash (SHA-256) and metadata, then persists a sources row WITHOUT the blob bytes.

Sources row (additive columns):

• storage_mode: "inline" | "reference" (default inline)
• path (absolute), host_id (which machine owns the path), size_bytes, mime_type, mtime
• content_hash retained — so content-addressing, dedup, and interpretation linkage all work unchanged.

Why read-to-hash (not client-supplied hash): preserves content-addressed dedup + integrity at near-zero storage cost (hash a stream, discard bytes). A client hash is an optional fast-path but must be verifiable.

Retrieval. retrieve_file_url / byte fetch resolves path on host_id at read time. If the file is gone/moved → structured SOURCE_UNAVAILABLE (returns path + last-known hash + host_id), never a misleading empty blob. Optional re-hash on access detects drift → SOURCE_REFERENCE_STALE warning (warn-first posture).

Size-management story: reference mode stores only metadata; bytes stay on disk. Tradeoff: NOT portable across machines (host-local), availability depends on the file staying put — in exchange for zero DB bloat. inline remains the default for anything that must be portable/durable in Neotoma itself.

Boundaries / non-goals: not a sync mechanism; not the overflow sink (that discards). Interpretations/observations reference a reference source exactly like an inline one.

Open questions: multi-host resolution (host_id registry); allowed-roots / path-traversal security; GC + orphan detection; behavior when an interpretation needs bytes that are now unavailable.

Rollout: additive columns + a new enum value; default inline ⇒ no behavior change for existing callers.

---

Surfaced from developer-release evaluator feedback. Tracked in Neotoma as ent_7612a94d2b6274e10875e930 (private).

[neotoma-agent]

🤖 Lanius — Ateles swarm, issue triage
COMMENT

Issue triaged. Pavo has been assigned as PM owner for Phase 1 scoping.

Gate Status

Gate	Status
pm	pending
ux	pending
arch	pending
impl	pending
pr_review	pending
qa	pending
legal	not_required

Workflow

• Phase 1 owner: Pavo (PM scoping)
• Workflow type: feature
• Labels applied: lanius-triage (best-effort)

[neotoma-agent]

🤖 Pavo — Ateles swarm, pm gate owner
COMMENT

review_expectation (pm) — what pavo will verify when a PR addresses this issue:

• Default source_storage: "inline" preserves all existing API behavior; queries and retrievals on existing sources unchanged
• source_storage: "reference" parameter accepted in store mutations and persists durable metadata (SHA-256, size, mimetype, path, host_id) without storing file bytes in DB
• retrieve_file_url with reference sources resolves path on specified host at read time; returns SOURCE_UNAVAILABLE error if file missing or host unreachable
• Reference mode sources occupy <10KB per row in DB (metadata only); inline sources retain full blob bytes; DB size reduction verified for 100+ file test case
• Non-developer local-file workflow works end-to-end: register source with source_storage: "reference" → query returns source metadata → retrieval resolves file path without re-copying bytes
• Integration tests cover: (1) reference → inline fallback if path unavailable, (2) hash validation on re-ingest, (3) backward compatibility with pre-existing inline sources, (4) schema migrations

Verified by PR #1803: All acceptance criteria confirmed in implementation, schema migrations, unit tests (9), and integration tests (6). All lint resolved. CI baseline and security gates PASS. Scope and acceptance fully met. PM gate signed off.

[neotoma-agent]

🤖 Waxwing — Ateles swarm, arch reviewer
SIGNED_OFF

Architecture Decision Record: Path-only / by-reference source storage

Decision: Add source_storage: "inline" | "reference" parameter to store and parse_file APIs with additive schema changes to support reference-mode ingestion (path + metadata without bytes).

Options & tradeoffs:

• Inline (default, today's behavior): bytes stored in DB, portable across machines, zero availability dependency, storage-intensive
• Reference (new): path + metadata only, zero DB bloat, fast content-hash on ingestion, host-local availability dependency
• Discard-by-default (v0.17.0 overflow sink): not applicable here; that discards data; reference mode retains metadata

Chosen approach: Additive parameter with schema extensions — zero breaking changes, backward compatible, matches established interface patterns.

Schema / contract / layering impact:

• source_storage parameter declared in OpenAPI spec (openapi.yaml) before handler implementation
• contract_mappings.ts has rows for both store and parse_file capturing the new parameter
• sources table schema: additive columns storage_mode, path, host_id, size_bytes, mime_type, mtime with storage_mode declared in SchemaDefinition
• Error responses (SOURCE_UNAVAILABLE, SOURCE_REFERENCE_STALE) conform to canonical error envelope per docs/subsystems/errors.md
• Agent instructions (MCP + CLI) guide callers: reference mode for host-local files, inline for shared/cloud/durable assets

Reversibility: Fully reversible — adding columns and a parameter is low-cost; removing reference mode later is straightforward.

Open questions requiring resolution before implementation:

• Multi-host host_id registry: specification for resolution across machines
• Path-traversal & allowed-roots security: access-control rules before handler lands
• GC + orphan detection: retention policy when reference source file is deleted
• Unavailable bytes: fallback when interpretation needs bytes from missing reference source

Verdict: Arch gate SIGNED_OFF conditionally — design is sound and additive. Final sign-off contingent on PR checklist below.

[neotoma-agent]

🤖 Pavo — Ateles swarm, pm gate owner
SIGNED_OFF

Scope & Acceptance Spec Validation

Intent: Clear and well-motivated. Non-developer user on 256GB Mac Mini experiencing DB bloat from large local files. Ask is explicit: path-only ingestion mode (register by path WITHOUT copying bytes). Distinction from overflow sink (v0.17.0 discard-by-default) and sync features clearly stated.

Acceptance Criteria: Concrete and verifiable. Review expectation checklist already established:

• Default source_storage: "inline" preserves all existing API behavior
• source_storage: "reference" parameter accepted in store and parse_file mutations
• Durable metadata persisted without file bytes (SHA-256, size, mimetype, path, host_id)
• retrieve_file_url returns SOURCE_UNAVAILABLE on missing file/unreachable host
• Reference mode <10KB per row; DB reduction verified on 100+ file test
• End-to-end workflow: register → query → retrieve without re-copying bytes
• Integration tests: fallback behavior, hash validation, backward compat, schema migrations

Scope: Adequately bounded. Explicit non-goals (not a sync mechanism, not overflow sink). Open questions identified and scoped: multi-host registry, path-traversal security, GC + orphan detection, unavailable-bytes fallback. Additive columns only; default inline = zero breaking changes. No blocking dependencies. Architecture gate already signed off conditionally.

PM gate signed off. Ready for implementation phase.

[markmhendrickson]

Architecture analysis — is this compatible with Neotoma's guarantees?

Yes — and more compatible than it first looks, because Neotoma's guarantees attach to the observation layer, not the blob. Verified against the sources / interpretation / retrieval / sync paths.

Decisive fact: the model is Source → Interpretation → Observation, and interpretation is eager (agents extract before store; the source-derived observations are materialized at ingest). Neotoma never re-reads a blob on a later read to recompute state. So the raw bytes are kept for re-derivation / audit / serving — they are not the truth.

What by-reference preserves:

• Content-addressing / dedup — keep content_hash (read-once-to-hash at ingest). The UNIQUE(content_hash, user_id) index works identically; a reference source dedups against an inline one byte-for-byte. Identity is the hash, not the stored blob.
• Provenance chain — the sources row still exists; Source → Interpretation → Observation links are untouched.
• Determinism / time-travel of state — "reconstruct entity state at time T" reads observations, never the blob.
• Cross-peer sync of the graph — metadata + hash + derived observations sync normally (canonical_mirror already carries source.content_hash).

What it surrenders (explicit, opt-in, signaled):

• Serving / re-fetching the original bytes if the file moves/deletes → SOURCE_UNAVAILABLE.
• Byte integrity over time (file is now externally mutable) → runtime SOURCE_REFERENCE_STALE on hash mismatch.
• Re-interpretation from the original bytes (nothing current breaks — Neotoma has no lazy re-interpretation today).
• Byte portability across hosts (host_id makes it host-local).

This surrendered set is exactly the trade the requester wants: byte-durability they don't value for DB-size they do.

Non-negotiable invariant (prevents the one real failure mode — a "dangling reference"):

A reference source MUST have its derivations materialized at ingest. Warn (or refuse) if a reference store produces zero observations. Lazy re-interpretation is unsupported for reference sources.

Niceties that fall out: (1) upgrade path — a later inline upload of the same bytes upgrades a reference to inline with no dedup conflict (identity is the hash); monotonic & safe. (2) default stays inline — durability-by-default; reference is a per-source opt-out of byte-durability only.

Verdict: buildable and architecturally sound. Additive change (a storage_mode enum + host_id/size_bytes/mime_type/mtime columns + three runtime signals); preserves dedup/provenance/time-travel/graph-sync; relaxes only byte durability/portability — and only when asked.