feat(docs): add docs sync system (#106)#107
Merged
Merged
Conversation
…106) Adds a CLI tool (pnpm sync:docs) that recursively syncs every .md file under docs/ into content_items as a note with source='docs-sync', tagged #project:shadowbrain, #docs, and a path-derived category tag. - src/lib/docs-sync.ts: syncDocsDirectory engine (idempotent upsert, tag self-healing, prune of deleted files, --force/--dry-run) - scripts/sync-docs.ts: CLI entry point mirroring import-markdown.ts - 27 tests covering idempotency, tags, cleanup, dry-run, force, and the hidden/private re-sync regression - version 0.8.0 -> 0.9.0 (new user-facing capability)
…timestamp display
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #106
What changed
Adds a docs sync system that keeps
docs/in sync with the ShadowBrain database, so docs are browsable, searchable, and available to chat from the web UI.src/lib/docs-sync.ts(new) —syncDocsDirectoryengine: recursively upserts every.mdfile under the sync root as anotewithsource='docs-sync'. Idempotent (skip-unchanged by default), prunes rows whose files were deleted from disk, and supports--force(re-write all) and--dry-run(preview, zero writes).scripts/sync-docs.ts(new) — CLI entry point mirroringimport-markdown.ts.src/lib/__tests__/docs-sync.test.ts(new) — 27 tests.package.json— addedsync:docsscript; version0.8.0→0.9.0(minor: new user-facing capability).README.md— version badge bumped in lockstep.Sync behavior
getting-started.md→getting-started).{ "file_path": "docs/<rel>" }.docs-sync-<sha256[:32]>— distinct prefix from notes (note-md-) so docs and notes sharing a relative path can't collide.#project:shadowbrain+#docs+ a category tag: top-level files →#docs:<stem>(getting-started.md→#docs:getting-started); nested files →#docs:<top-dir>(api/endpoints/auth.md→#docs:api). Tags self-heal viaINSERT OR IGNOREeven on skipped files.source='docs-sync'; the prune audit log records thefile_path(the id alone is an opaque hash).Usage:
How verified
pnpm verifygreen: lint ✓, typecheck ✓, build ✓,pnpm test --run719/719 ✓ (27 in the new docs-sync suite), knip ✓.docs/: 36 docs discovered → 36 created; re-run → 36 skipped (idempotent); category tags verified correct across nested and top-level paths.@oracle review
Two-pass strategic + security review (triggered: diff touches the database layer). No MUST-FIX blockers. All 5 SHOULD-FIX findings addressed before opening:
minor(0.9.0) per AGENTS.md §Versioning.parseMarkdownFilecall (title computed inline; avoids spurious frontmatter warnings).generateDocsIddocstring.Plus 3 nits addressed (self-heal test asserts tag identity, softened two overclaiming docstrings).
Intentionally deferred (out of scope for this PR)
audit_logsrow for skipped / too-large files — parity with the existing markdown importer, which also writes none.walkMarkdownFiles/isHiddenPathduplicated from the markdown importer — extracting a shared module is unrelated churn.ensureTagis not race-free against concurrent syncs — this is a single-operator CLI with no concurrent writers.