Expand nostrdb docs & mdbook

.gitignore

@@ -39,3 +39,7 @@ ndb
 
 /target
 /Cargo.lock
+
+# mdBook output
+/book
+/docs/book/build

README.md

@@ -18,6 +18,95 @@ as a C library into any application with full nostr query support.
 
 [1]: https://github.com/hoytech/strfry
 
+## Why nostrdb
+
+- Zero-copy note model backed by [LMDB](https://symas.com/lmdb/) so queries simply point at memory-mapped pages.
+- Custom metadata and block formats give O(1) access to note fields, reactions, and parsed blocks.
+- Built-in full text indexing plus CLI tooling (`ndb`) for ad-hoc inspection.
+- Ships as a portable static library (`libnostrdb.a`) with bindings for C, Rust, and Swift.
+- Focused on embeddability—drop it into a relay, client, or analytics pipeline.
+
+## Project status
+
+The API is **very unstable**. nostrdb is in heavy development mode so expect
+breaking changes between commits. Track higher level docs in `docs/` for the latest APIs.
+
+## Supported platforms & dependencies
+
+nostrdb targets modern Unix-like systems (Linux, macOS). Windows builds require additional
+work and are not covered by the default build. The repository vendors most dependencies but
+you still need a toolchain with:
+
+- A C11 compiler (tested with Clang 15+ and GCC 12+)
+- `make`, `pkg-config`, and standard build tools (autoconf/automake for secp256k1)
+- `curl` and `zstd` (used to download sample data)
+- Optional: `flatc`/`flatcc` if regenerating schema bindings, `nix` if using `shell.nix`
+
+Third-party libraries are fetched/built automatically:
+
+- [LMDB](https://github.com/LMDB/lmdb) (memory-mapped storage)
+- [libsecp256k1](https://github.com/bitcoin-core/secp256k1) (signature verification)
+- [flatcc](https://github.com/dvidelabs/flatcc) / [flatbuffers](https://github.com/google/flatbuffers) (schema tooling)
+- [ccan](https://github.com/rustyrussell/ccan) utilities
+
+## Quick start
+
+```bash
+git clone https://github.com/damus-io/nostrdb.git
+cd nostrdb
+# refresh vendored submodules (libsecp256k1)
+./devtools/refresh-submodules.sh deps/secp256k1
+
+# build the CLI + static library
+make ndb   # or simply `make` to build ndb, libnostrdb.a, and benchmarks
+```
+
+### Run the sanitizer-backed tests
+
+```bash
+make testdata/db/.dir        # ensures the temporary LMDB dir exists
+make test                    # builds with ASan/UBSan flags and runs ./test
+```
+
+### Import sample data and query
+
+```bash
+# Download fixtures (zstd archives are pulled from jb55.com CDN)
+make testdata/many-events.json
+
+# Create an empty LMDB environment in ./data (default) and ingest
+./ndb --skip-verification import testdata/many-events.json
+
+# Run a text search (newest first by default)
+./ndb query --search "nostrdb" --limit 5
+
+# Print global stats about the database
+./ndb stat
+```
+
+Extra fixtures are listed under `testdata/` (see Makefile targets). You can also point `ndb`
+at an existing relay database via `-d path/to/db`.
+
+### Where to go next
+
+- `docs/getting-started.md` – detailed environment setup, workflows, and a minimal C ingestion example.
+- `docs/architecture.md` – how LMDB, note blocks, metadata, and indexes fit together.
+- `docs/api.md` – tour of the public C API surfaces.
+- `docs/cli.md` – comprehensive CLI command reference.
+- `docs/bindings/index.md` – pointers for C, Rust, and Swift consumers.
+
+### Render the docs as a book
+
+The `docs/book` directory contains an [mdBook](https://rust-lang.github.io/mdBook/) configuration
+that stitches all documentation together into a browsable site:
+
+```bash
+cargo install mdbook        # once, if you don't already have it
+mdbook serve docs/book --open
+```
+
+The `serve` task watches for file changes and rebuilds automatically. Use `mdbook build docs/book`
+to emit static HTML under `docs/book/build/`.
 
 ## API
 

docs/api.md

@@ -0,0 +1,140 @@
+# API Tour
+
+This document summarizes the primary entry points exposed by `src/nostrdb.h`.
+It is not an exhaustive reference but should help you find the right family of
+functions for common tasks. See the header for full signatures and structure
+definitions.
+
+## Configuration & lifecycle
+
+```c
+struct ndb *db;
+struct ndb_config config;
+ndb_default_config(&config);
+ndb_config_set_mapsize(&config, 1ULL << 34); // 16 GiB
+ndb_config_set_flags(&config, NDB_FLAG_SKIP_NOTE_VERIFY);
+ndb_config_set_ingest_threads(&config, 4);
+ndb_config_set_ingest_filter(&config, my_filter, ctx);
+ndb_config_set_subscription_callback(&config, my_sub_cb, ctx);
+ndb_config_set_writer_scratch_buffer_size(&config, 4 * 1024 * 1024);
+
+if (!ndb_init(&db, "./data", &config)) { /* handle error */ }
+...
+ndb_destroy(db);
+```
+
+- `ndb_default_config` – zeroes + sensible defaults.
+- `ndb_config_set_*` – tune ingest threads, flags, LMDB map size, filters, subscription callbacks.
+- `ndb_init` – opens/creates the LMDB environment at `dbdir`. Always pair with `ndb_destroy`.
+
+## Ingestion helpers
+
+- `ndb_process_event`, `ndb_process_events` – parse nostr JSON (single event or LDJSON batch).
+- `ndb_process_event_with`, `ndb_process_events_with` – include relay/client metadata (`struct ndb_ingest_meta`).
+- `ndb_process_events_stream` – stream reader-friendly variant used by the CLI.
+- `ndb_ingest_meta_init` – convenience initializer for metadata.
+- `ndb_process_client_event(s)` – legacy wrappers (prefer the `ndb_process_event*` family).
+
+### Metadata builders
+
+Metadata is created/updated via `ndb_note_meta_builder`:
+
+```c
+unsigned char buf[1024];
+struct ndb_note_meta_builder builder;
+ndb_note_meta_builder_init(&builder, buf, sizeof buf);
+
+struct ndb_note_meta_entry *entry = ndb_note_meta_add_entry(&builder);
+ndb_note_meta_reaction_set(entry, /*count=*/5, reaction);
+
+struct ndb_note_meta *meta;
+ndb_note_meta_build(&builder, &meta);
+ndb_set_note_meta(db, note_id, meta);
+```
+
+Key functions:
+
+- `ndb_note_meta_builder_init`, `ndb_note_meta_builder_resized`, `ndb_note_meta_build`
+- `ndb_note_meta_add_entry`, `ndb_note_meta_builder_find_entry`
+- Accessors such as `ndb_note_meta_counts_*`, `ndb_note_meta_reaction_set`, `ndb_note_meta_flags`
+- `ndb_get_note_meta`, `ndb_note_meta_entries`, `ndb_note_meta_find_entry`
+
+## Transactions & queries
+
+Readers borrow LMDB transactions via `ndb_begin_query` / `ndb_end_query`.
+
+```c
+struct ndb_txn txn;
+if (!ndb_begin_query(db, &txn)) { /* handle error */ }
+
+struct ndb_filter filter;
+ndb_filter_init(&filter);
+ndb_filter_start_field(&filter, NDB_FILTER_KINDS);
+ndb_filter_add_int_element(&filter, 1);
+ndb_filter_end_field(&filter);
+ndb_filter_end(&filter);
+
+struct ndb_query_result results[32];
+int count = 0;
+ndb_query(&txn, &filter, 1, results, 32, &count);
+
+for (int i = 0; i < count; i++) {
+    struct ndb_note *note = results[i].note;
+    printf("kind=%u content=%.*s\n",
+           ndb_note_kind(note),
+           ndb_note_content_length(note),
+           ndb_note_content(note));
+}
+
+ndb_filter_destroy(&filter);
+ndb_end_query(&txn);
+```
+
+Notable APIs:
+
+- `ndb_filter_init`, `ndb_filter_init_with` – allocate filter buffers.
+- `ndb_filter_start_field`, `ndb_filter_start_tag_field`, `ndb_filter_add_*` – build queries.
+- `ndb_filter_matches*`, `ndb_filter_is_subset_of`, `ndb_filter_clone`, `ndb_filter_json`.
+- `ndb_filter_from_json` – decode NIP-01 filters.
+- `ndb_query` – execute filter arrays.
+- `ndb_subscribe`, `ndb_wait_for_notes`, `ndb_poll_for_notes`, `ndb_unsubscribe` – pub/sub APIs driven by ingest callbacks.
+
+### Text search
+
+- `ndb_text_search` / `ndb_text_search_with` – ranked searches over `NDB_DB_NOTE_TEXT`.
+- `ndb_default_text_search_config`, `ndb_text_search_config_set_order`, `ndb_text_search_config_set_limit` – configure sort order and result caps.
+
+## Profiles, relays, and helpers
+
+- `ndb_search_profile`, `ndb_search_profile_next`, `ndb_search_profile_end` – profile text search.
+- `ndb_get_profile_by_pubkey`, `ndb_get_profile_by_key`, `ndb_get_profilekey_by_pubkey` – profile cache access.
+- `ndb_get_notekey_by_id`, `ndb_get_note_by_id`, `ndb_get_note_by_key` – note lookups.
+- `ndb_note_seen_on_relay`, `ndb_note_relay_iterate_start`, `ndb_note_relay_iterate_next` – relay tracking.
+- `ndb_write_last_profile_fetch`, `ndb_read_last_profile_fetch` – per-pubkey sync bookkeeping.
+
+Helper utilities:
+
+- `ndb_create_keypair`, `ndb_decode_key`, `ndb_sign_id`, `ndb_calculate_id`, `ndb_note_verify`
+- `ndb_parse_content`, `ndb_blocks_iterate_*`, `ndb_blocks_free` – block parser helpers for rendering.
+
+## Note & tag inspection
+
+Once you have `struct ndb_note *` pointers:
+
+- `ndb_note_id`, `ndb_note_pubkey`, `ndb_note_sig`, `ndb_note_kind`, `ndb_note_created_at`
+- `ndb_note_content`, `ndb_note_content_length`, `ndb_note_json`
+- `ndb_note_tags`, `ndb_tags_iterate_start`, `ndb_tags_iterate_next`, `ndb_iter_tag_str`
+- `ndb_note_meta_iterator`, `ndb_note_meta_entry_at`, `ndb_note_meta_reaction_str`
+
+These functions operate on the packed, zero-copy note representation without heap
+allocations.
+
+## Error handling tips
+
+- Most APIs return `int` (non-zero success). Check return codes and call
+  `ndb_destroy` on failure to release LMDB handles.
+- Filters must be finalized with `ndb_filter_end` before calling `ndb_query`.
+- Reader transactions must be closed (`ndb_end_query`) before destroying the DB.
+
+For deeper explanations of the storage layers see `docs/architecture.md`. For CLI
+examples and manual test workflows refer to `docs/cli.md` and `docs/getting-started.md`.

docs/architecture.md

@@ -0,0 +1,124 @@
+# Architecture
+
+nostrdb is an embeddable storage engine tailored for nostr events. It adopts the
+LMDB single-writer/multi-reader model and layers a custom in-memory note format,
+metadata tables, and indexes on top of LMDB key/value pairs. This document
+summarizes the moving pieces so you can reason about performance and extend the
+system safely.
+
+## High-level data flow
+
+```
+ JSON event --> parser/builder --> packed note + metadata --> LMDB buckets
+                                                   |
+                                    derived indexes (tags, relays, text, ...)
+```
+
+1. **Ingestion** – `ndb_process_event*` (see `src/nostrdb.c`) parses JSON into
+   a compact `struct ndb_note` using cursor-backed builders (`struct ndb_builder`
+   in `src/nostrdb.h`). Optional ingest metadata (relay, client) gets recorded.
+2. **Validation** – Signatures are checked via `libsecp256k1` unless
+   `NDB_FLAG_SKIP_NOTE_VERIFY` is set in the config.
+3. **Persistence** – The packed note, associated metadata (`struct ndb_note_meta`),
+   and derived indexes are written into an LMDB environment using the single writer thread.
+4. **Query** – Readers open LMDB read transactions (`struct ndb_txn`), memory-map note
+   blobs, and evaluate filters or full-text queries without copying data out of LMDB.
+
+## LMDB layout
+
+The database uses multiple LMDB databases (`enum ndb_dbs` in `src/nostrdb.h`), the most
+important being:
+
+- `NDB_DB_NOTE` – primary storage for packed notes
+- `NDB_DB_META` – note metadata (reactions, counts)
+- `NDB_DB_PROFILE` / `NDB_DB_PROFILE_PK` – cached profiles and pubkey index
+- `NDB_DB_NOTE_TEXT` – full-text search index
+- `NDB_DB_NOTE_KIND`, `NDB_DB_NOTE_TAGS`, `NDB_DB_NOTE_PUBKEY[_KIND]` – secondary indexes
+- `NDB_DB_NOTE_RELAYS` / `NDB_DB_NOTE_RELAY_KIND` – relay tracking
+
+All buckets live inside the same LMDB environment and benefit from LMDB’s MVCC
+semantics: readers see a consistent view, the single writer thread avoids locks, and
+data is accessed by memory-mapping the LMDB pages.
+
+## Packed notes & metadata
+
+`struct ndb_note` packs the event ID, pubkey, signature, created-at, kind, tags, and
+content into cursor-managed buffers (see `struct ndb_builder`). Strings are either
+zero-copy references or 32-byte IDs flagged with `NDB_PACKED_STR`/`NDB_PACKED_ID`.
+
+Metadata lives alongside the note as a table of 16-byte entries
+(`struct ndb_note_meta_entry`, documented in `docs/metadata.md`). Each entry stores:
+
+- a type (counts, reactions, thread state, custom odd-numbered tags)
+- an 8-byte payload (counts or offsets into a data table)
+- optional aux data (e.g., total reaction counts)
+
+The format keeps frequently updated counters (direct replies, quotes, reactions)
+in-place, enabling atomic updates without decoding/recoding blobs.
+
+## Ingestion pipeline
+
+1. **Parsing** – Notes are parsed via `ndb_note_from_json` or the CLI’s streaming
+   helpers. The parser feeds an `ndb_builder` that writes into scratch buffers sized
+   by `ndb_config_set_writer_scratch_buffer_size`.
+2. **Filtering** – `ndb_config_set_ingest_filter` allows applications to inspect
+   notes before they hit disk (accept/reject/skip validation).
+3. **Metadata augmentation** – Reactions, relay sightings, and thread stats are
+   updated via metadata builder helpers (`ndb_note_meta_builder_*`).
+4. **Index maintenance** – As the single writer thread commits, it updates the
+   secondary indexes (tag, relay, kind, pubkey, full-text) and persists metadata.
+
+Because LMDB enforces a single writer, nostrdb runs the ingestion path on a
+dedicated thread. Reader-heavy workloads scale because reads are lock-free and
+operate on memory-mapped pages.
+
+## Query path
+
+Readers call `ndb_begin_query` to open an LMDB read transaction and construct one
+or more filters (`struct ndb_filter`). Filters can match IDs, authors, kinds,
+tags (`#e`, `#p`, custom), time ranges, search tokens, relays, or custom callbacks.
+
+`ndb_query` takes a filter array, executes it against indexes, and returns a list
+of `struct ndb_query_result` entries with direct `struct ndb_note *` pointers; no
+copying occurs because LMDB pages remain mapped for the lifetime of the transaction.
+
+Full-text searches (`ndb_text_search*`) use the same reader transactions but consult
+`NDB_DB_NOTE_TEXT` for ranked matches. Results can be fed through filters to enforce
+kind/author/relay constraints.
+
+## Blocks, content parsing, and relays
+
+`src/content_parser.c` and related block structures (`struct ndb_blocks`,
+`struct ndb_block_iterator`) parse note content into blocks (text, mentions,
+invoices, bech32 references) for rich rendering. Parsed blocks are cached in
+`NDB_DB_NOTE_BLOCKS` to avoid reparsing long-form notes.
+
+Relays are tracked with iterators (`struct ndb_note_relay_iterator`) that read from
+`NDB_DB_NOTE_RELAYS` and can be used both by the CLI (`ndb note-relays`) and
+embedders implementing gossip logic.
+
+## CLI + bindings
+
+The `ndb` CLI (`ndb.c`) exercises each subsystem:
+
+- `stat` – reads global LMDB stats (`ndb_stat`)
+- `query` – builds filters from CLI flags and prints JSON notes
+- `import` – streams line-delimited JSON into `ndb_process_events`
+- `profile`, `note-relays`, `print-*` – showcase profile cache, relay iterators,
+  search indexes, and metadata printing helpers
+
+Bindings in `src/bindings/{c,rust,swift}` expose the same primitives to higher-level
+languages. They are generated from `schemas/*.fbs` via `flatcc`/`flatc`.
+
+## Extending nostrdb safely
+
+- Prefer new LMDB buckets over overloading existing ones; update `enum ndb_dbs` and
+  provide readable names via `ndb_db_name`.
+- Keep ingestion work inside the writer thread. Use metadata builders for counters to
+  avoid race conditions.
+- Ensure filters are always finalized (`ndb_filter_end`) before executing queries.
+- When adjusting metadata formats, bump version fields in `struct ndb_note_meta` so
+  older readers can detect incompatibilities.
+
+For concrete API references, see `docs/api.md`. For CLI usage patterns, refer to
+`docs/cli.md`.