docs: expand architecture and persistence guides

Author: SepineTam

docs/en/architecture.md

@@ -0,0 +1,84 @@
+# System Architecture
+
+NBER-CLI is organized as a small layered application. The command line interface and the MCP server are entry points; the shared core performs network retrieval, parsing, formatting, downloads, feed processing, and local persistence.
+
+## Component Map
+
+```mermaid
+flowchart TD
+    user[CLI user] --> cli[cli.py]
+    agent[AI agent] --> mcp[mcp.py / FastMCP]
+    cli --> fetcher[fetcher.py]
+    cli --> download[download.py]
+    cli --> feed[feed.py]
+    cli --> cache[info_cache.py]
+    mcp --> fetcher
+    mcp --> download
+    mcp --> cache
+    fetcher --> nber[NBER web pages and search API]
+    download --> pdf[NBER PDF endpoint]
+    feed --> rss[NBER RSS feed]
+    cache --> db[db.py / SQLite]
+    feed --> db
+    cli --> db
+    db --> config[config_store.py / config.json]
+```
+
+## Entry Points
+
+| Surface | File | Main role |
+| --- | --- | --- |
+| Console script | `src/nber_cli/cli.py` | Parses arguments, prints text or JSON, records CLI logs, and maps user commands to shared functions. |
+| Python module | `src/nber_cli/__main__.py` | Lets users run `python -m nber_cli` with the same behavior as `nber-cli`. |
+| MCP server | `src/nber_cli/mcp.py` | Exposes `get_paper_info`, `search_papers`, and `download_paper` for agent clients. |
+| Public package API | `src/nber_cli/__init__.py` | Defines the top-level stable imports through `__all__`. |
+
+## Core Workflows
+
+| Workflow | Path through the system | Persistence behavior |
+| --- | --- | --- |
+| Search | `cli.py` or `mcp.py` -> `fetcher.search_nber` -> NBER search API -> `formatters.search_results` | CLI search records `query_log`; MCP search does not. |
+| Paper info | `cli.py` or `mcp.py` -> `info_cache.py` -> `db.py` cache lookup -> `fetcher.get_nber` on miss | Reads and writes `info_cache` when enabled; both surfaces record `info_log`. |
+| Download | `cli.py` or `mcp.py` -> `download.py` -> NBER PDF endpoint | CLI download records `download_log`; MCP download does not. |
+| Feed | `cli.py` -> `feed.py` -> NBER RSS -> `db.py` | Stores feed items in `feed_items` and fetch summaries in `feed_fetches`. |
+| Config | `cli.py` -> `config_store.py` | Reads and writes `~/.nber-cli/config.json`. |
+
+## Network Layer
+
+`fetcher.py` retrieves paper pages and search results. It sends browser-like headers for every NBER request, enforces TLS 1.2 for synchronous page loads, retries transient failures, and validates that a fetched paper page matches the requested paper ID.
+
+`download.py` uses `aiohttp` for PDF downloads. Single downloads buffer the full PDF in memory before writing to disk. Batch downloads share a client session and use an `asyncio.Semaphore` to cap concurrency.
+
+`feed.py` fetches the public RSS feed, parses it with `defusedxml`, repairs a narrow class of malformed text containing unescaped `<` characters, and ignores malformed items that cannot produce a paper ID.
+
+## Output and Formatting
+
+The CLI keeps the human-readable output separate from the structured payloads:
+
+- `formatters.info` and `formatters.info_text` format paper metadata.
+- `formatters.search_results` and `formatters.search_results_text` format search results.
+- `formatters.feed_results` and `formatters.feed_results_text` format RSS feed results.
+
+The MCP server returns dictionaries rather than CLI text. This lets agents consume the same underlying data without scraping terminal output.
+
+## Trust Boundaries
+
+NBER-CLI does not require credentials and does not send the local database to project infrastructure. The risky boundary is filesystem writes:
+
+- CLI downloads are restricted to the current directory by default, but users can pass `--restrict false`.
+- MCP downloads always normalize the paper ID, restrict writes to the server process working directory, and return an error for paths outside that directory.
+- HTTP MCP transport has no built-in authentication. Treat it as local-only unless it is placed behind an authenticating proxy or tunnel.
+
+## Source-to-Concept Reference
+
+| Concept | Primary files |
+| --- | --- |
+| CLI command model | `src/nber_cli/cli.py` |
+| Agent tool model | `src/nber_cli/mcp.py` |
+| Public Python API | `src/nber_cli/__init__.py`, `docs/en/python-api.md` |
+| Search and metadata parsing | `src/nber_cli/fetcher.py` |
+| PDF download engine | `src/nber_cli/download.py` |
+| RSS feed system | `src/nber_cli/feed.py` |
+| Info cache | `src/nber_cli/info_cache.py`, `src/nber_cli/db.py` |
+| Local config | `src/nber_cli/config_store.py`, `src/nber_cli/config.schema.json` |
+| Local database | `src/nber_cli/db.py` |

docs/en/configuration.md

@@ -2,6 +2,8 @@
 
 Most NBER-CLI runtime behavior uses built-in defaults. The local database also uses a small user config file to remember the database location selected by `nber-cli db init` or `nber-cli db migrate`.
 
+For the full SQLite schema, cache tables, behavior logs, and backup guidance, see [Persistence Layer](persistence.md).
+
 ## Runtime Defaults
 
 | Setting | Default | Description |

docs/en/development.md

@@ -2,6 +2,8 @@
 
 This page covers local development, testing, documentation, and release preparation.
 
+For the detailed fixture and mocking layout, see [Test Infrastructure](testing.md). For runtime component relationships, see [System Architecture](architecture.md).
+
 ## Repository Layout
 
 ```text

docs/en/glossary.md

@@ -0,0 +1,75 @@
+# Glossary
+
+## NBER Paper ID
+
+A working paper identifier such as `w25000`. Most user-facing commands accept the ID with or without the `w` prefix, but URLs and default PDF filenames use the canonical `w` prefix.
+
+## CLI
+
+The `nber-cli` command implemented in `src/nber_cli/cli.py`. It is optimized for humans and scripts: text output by default, JSON output where supported, and stable non-zero exits for failures.
+
+## MCP Server
+
+The agent-facing server implemented in `src/nber_cli/mcp.py` with FastMCP. It exposes three tools: `get_paper_info`, `search_papers`, and `download_paper`.
+
+## Info Cache
+
+The `info_cache` table and `info_cache.py` helper flow. It stores paper metadata so repeated `info` and `get_paper_info` calls can avoid another NBER page fetch while the row is within the configured TTL.
+
+## Sliding TTL
+
+The info cache refresh model. On a cache hit, NBER-CLI updates `last_fetched_at` and increments `fetch_count`, so expiration is measured from the latest local hit rather than only from the original network fetch.
+
+## Feed System
+
+The RSS workflow in `feed.py`. It reads `https://www.nber.org/rss/new.xml`, parses items with `defusedxml`, stores them in `feed_items`, and returns new or all items depending on the command options.
+
+## Local Database
+
+The SQLite database resolved by `db.get_database_path()`. By default it lives at `~/.nber-cli/nber.db`. It stores feed cache rows, info cache rows, and behavior logs.
+
+## Config File
+
+The JSON file at `~/.nber-cli/config.json`. It stores the configured database path, database schema version, info cache settings, and download defaults.
+
+## SQLModel
+
+The typed ORM layer used by `db.py` on top of SQLAlchemy. Tables such as `FeedItem`, `InfoCache`, `QueryLog`, `DownloadLog`, and `InfoLog` are declared as SQLModel models.
+
+## Behavior Logs
+
+Local-only tables that record operational events:
+
+- `query_log` records CLI search keywords and result counts.
+- `download_log` records CLI download outcomes.
+- `info_log` records CLI and MCP paper info lookups.
+
+These logs are not sent to any project server.
+
+## Soft Failure
+
+A non-critical database write failure that should not break the main operation. For example, a failed `query_log` write should not prevent search results from being printed.
+
+## Download Restriction
+
+The filesystem safety rule that keeps download targets inside the current working directory by default. The CLI can override this with `--restrict false`; MCP downloads always enforce the restriction.
+
+## `display_all`
+
+A feed fetch option. When false, `feed fetch` displays only newly discovered RSS entries. When true, it displays all currently fetched RSS entries, including rows already present in the cache.
+
+## `include_all`
+
+An MCP `get_paper_info` option. When true, the returned dictionary includes related fields and `published_version` when NBER exposes them.
+
+## `--refresh`
+
+A CLI `info` flag that bypasses the local info cache for one call and fetches the paper page from NBER again. MCP does not currently expose an equivalent per-call flag.
+
+## First-Week Restriction
+
+NBER can return HTTP 403 for newly released papers that are not yet publicly downloadable. NBER-CLI reports this as a permission/access message rather than treating it as a missing paper.
+
+## Future Schema
+
+A database whose SQLite `PRAGMA user_version` is newer than the version supported by the installed NBER-CLI. The package rejects writes to this database to avoid downgrade corruption.

docs/en/index.md

@@ -63,10 +63,14 @@ Add it to an MCP client:
 - [MCP Server](mcp.md): agent configuration, transports, and available tools.
 - [Python API](python-api.md): async functions and data models.
 - [Configuration](configuration.md): runtime defaults and operational behavior.
+- [Persistence Layer](persistence.md): local SQLite schema, cache behavior, logs, migration, and cleanup limits.
+- [System Architecture](architecture.md): how the CLI, MCP server, network layer, download engine, feed system, and persistence layer fit together.
+- [Glossary](glossary.md): project-specific terms, table names, error models, and paper ID conventions.
 - [Development](development.md): local setup, tests, docs, CI, and release workflow.
+- [Test Infrastructure](testing.md): fixtures, mocking strategy, async tests, and robustness coverage.
 - [Contributing](contributing.md): contribution standards and review expectations.
 - [Changelog](changelog.md): notable project changes.
 
 ## Project Status
 
-The current public command model is `nber-cli` v0.3.0. The CLI is intentionally small and script-friendly: text output is optimized for human reading, and `--format json` is available where structured output matters.
+The current public command model is `nber-cli` v0.7.0. The CLI is intentionally small and script-friendly: text output is optimized for human reading, and `--format json` is available where structured output matters.

docs/en/mcp.md

@@ -2,6 +2,8 @@
 
 NBER-CLI includes an MCP server so agents can search NBER, inspect paper metadata, and download PDFs without scraping command output.
 
+For how the MCP server shares core modules with the CLI, see [System Architecture](architecture.md).
+
 ## Start the Server
 
 The default transport is stdio:

docs/en/persistence.md

@@ -0,0 +1,100 @@
+# Persistence Layer
+
+NBER-CLI uses a local SQLite database plus a JSON config file. The database stores caches and operational logs; the config file stores user-selected runtime settings such as the database path, info cache behavior, and download defaults.
+
+## Files
+
+| File | Default path | Purpose |
+| --- | --- | --- |
+| Config | `~/.nber-cli/config.json` | Stores `schema_version`, `feed.db-path`, `info` cache settings, and `download` settings. |
+| Database | `~/.nber-cli/nber.db` | Stores feed cache, paper info cache, and behavior logs. |
+| Legacy database | `~/.nber-cli/feed.db` | Used only as a fallback when upgrading from older releases and no `nber.db` exists. |
+| Debug log | `~/.nber-cli/debug.log` | Rotating log file for warnings, errors, and debug output when enabled. |
+
+## Database Tables
+
+| Table | Primary use | Written by | Cleaned by |
+| --- | --- | --- | --- |
+| `feed_items` | Cached RSS paper entries keyed by paper ID. | `feed fetch` | `feed clean` |
+| `feed_fetches` | Audit trail for RSS fetch attempts and counts. | `feed fetch` | No CLI cleanup command. |
+| `info_cache` | Cached paper metadata used by `info` and `get_paper_info`. | `info`, MCP `get_paper_info` | `info cache clear` |
+| `query_log` | CLI search query history and result counts. | CLI `search` | No CLI cleanup command. |
+| `download_log` | CLI download successes and failures. | CLI `download` | No CLI cleanup command. |
+| `info_log` | Paper info lookup history. | CLI `info`, MCP `get_paper_info` | No CLI cleanup command. |
+
+The schema version is stored in SQLite `PRAGMA user_version`. Version `2` is the current schema. NBER-CLI refuses to write to a database created by a newer schema version.
+
+## Info Cache Behavior
+
+The info cache is controlled by:
+
+```json
+{
+  "info": {
+    "cache_enabled": true,
+    "cache_ttl_days": 30
+  }
+}
+```
+
+When the cache is enabled, `info` and MCP `get_paper_info` first check `info_cache`. A fresh cache hit returns local data and then updates `last_fetched_at` plus `fetch_count`. This makes the TTL sliding: frequently used paper metadata remains fresh relative to the most recent local hit.
+
+Use `--refresh` on the CLI to bypass the cache for one `info` call:
+
+```bash
+nber-cli info w25000 --refresh
+```
+
+The MCP tool does not expose a per-call refresh flag. To force a live MCP lookup, disable the cache temporarily or wait for the TTL to expire.
+
+## Feed Cache Behavior
+
+`feed fetch` stores every fetched RSS item, then returns either only newly discovered items or all fetched items:
+
+```bash
+nber-cli feed fetch
+nber-cli feed fetch --display-all true
+nber-cli feed fetch --max-items 5
+```
+
+`feed_items` is keyed by paper ID. Existing rows are updated with the latest title, abstract, URL, source URL, GUID, authors, and `last_seen_at`. New rows keep their original `first_seen_at`.
+
+`feed_fetches` is an append-only audit table. `feed clean` does not remove it.
+
+## Logs and Soft Failures
+
+Search, download, and info operations try to append behavior logs. These writes are intentionally non-critical: a database error can emit a warning, but it should not prevent the main search, download, or metadata lookup from completing.
+
+Cache reads also fail soft where possible. If a cache read cannot be completed safely, the command falls back to the network path or returns an empty cache count depending on the helper.
+
+## Migration and Path Rules
+
+Initialize or move the database:
+
+```bash
+nber-cli db init
+nber-cli db init --db-path ~/data/nber.db
+nber-cli db init --db-path sqlite:////Users/name/data/nber.db
+nber-cli db migrate ~/data/nber.db
+```
+
+On macOS and Linux, the database path must stay inside the user's home directory. This limit avoids accidental writes into system or shared locations. The destination for `db migrate` must not already exist, and sidecar files such as `-wal`, `-shm`, and `-journal` move with the database.
+
+## Cleanup Coverage
+
+```bash
+nber-cli feed clean --days 30
+nber-cli feed clean --all
+nber-cli info cache clear --days 30
+nber-cli info cache clear --all
+```
+
+Both cleanup commands show a preview and require confirmation before deleting rows. `feed clean` deletes only `feed_items`. `info cache clear` deletes only `info_cache`. Logs and `feed_fetches` require manual SQLite maintenance or a fresh database.
+
+## Backup
+
+For a safe backup, stop any running CLI or MCP process and copy `nber.db` together with any `nber.db-wal` and `nber.db-shm` sidecar files. For live systems, use SQLite's backup command:
+
+```bash
+sqlite3 ~/.nber-cli/nber.db ".backup '/path/to/nber-backup.db'"
+```

docs/en/testing.md

@@ -0,0 +1,81 @@
+# Test Infrastructure
+
+The test suite is built with Pytest and is designed around isolation. Tests avoid writing into the real home directory, avoid live NBER network calls, and cover both CLI and library behavior.
+
+## Run Tests
+
+```bash
+uv run pytest
+uv run pytest tests/test_cli.py
+uv run pytest -m "not slow"
+```
+
+Run linting and documentation checks with:
+
+```bash
+uv run ruff check .
+uv run --group docs mkdocs build --strict
+```
+
+## Test Suite Map
+
+| Area | Representative files | What it covers |
+| --- | --- | --- |
+| CLI | `tests/test_cli.py`, `tests/test_main.py` | Argument parsing, subcommand behavior, output formats, exit behavior. |
+| Network fetcher | `tests/test_fetcher.py` | Paper page parsing, search payload parsing, retry and request behavior. |
+| Downloads | `tests/test_downloader.py` | Single and batch download paths, validation, failures, concurrency behavior. |
+| Feed | `tests/test_feed.py` | RSS parsing, malformed XML handling, new-item detection, cleanup. |
+| Database | `tests/test_db.py`, `tests/test_config_store.py` | Schema creation, config persistence, migration, path normalization, cache tables. |
+| Info cache | `tests/test_info_cache.py`, `tests/test_info_cache_flow.py` | Cache hits, refresh behavior, TTL logic, integration with `info`. |
+| MCP | `tests/test_mcp.py` | Tool return shapes, error handling, paper ID normalization, download path restrictions. |
+| Logging | `tests/test_logging.py`, `tests/test_logs.py` | Log configuration, debug behavior, rotating file setup. |
+
+## Isolation Model
+
+The global fixture in `tests/conftest.py` redirects the NBER-CLI home directory behavior into a temporary path. This protects the user's real `~/.nber-cli/config.json`, database, and debug log during test runs.
+
+Tests patch `Path.home()`, database paths, network functions, and HTTP sessions as needed. The goal is that a test can be run repeatedly without depending on the developer's machine state or network access.
+
+```mermaid
+flowchart LR
+    pytest[Pytest] --> fixture[isolated_nber_home]
+    fixture --> temp[temporary home]
+    temp --> config[config.json]
+    temp --> database[nber.db]
+    pytest --> mocks[network and session mocks]
+    mocks --> fetcher[fetcher.py]
+    mocks --> download[download.py]
+    mocks --> feed[feed.py]
+```
+
+## Mocking Strategy
+
+Network-facing tests mock the lowest practical boundary:
+
+- Synchronous page and feed retrieval patch `_load_text_sync` or `urllib.request.urlopen`.
+- Async search and download paths use fake `aiohttp` sessions or async mocks.
+- CLI tests patch high-level functions when the test is about argument dispatch rather than parsing NBER responses.
+
+This split keeps parser tests focused on parsing, command tests focused on CLI behavior, and integration-style tests focused on component interaction.
+
+## Async Patterns
+
+Async functions are tested through `pytest-asyncio` or by invoking CLI paths that internally call `asyncio.run`. Download batch tests assert both successful paths and collected failures because `download_multiple_papers` returns a `DownloadBatchResult` instead of raising when individual downloads fail.
+
+## Robustness Coverage
+
+The tests intentionally cover edge cases that are easy to break:
+
+- Paper IDs with and without the `w` prefix.
+- Invalid paper IDs and mismatched fetched paper IDs.
+- Search pagination limits and date defaults.
+- XML entities and malformed RSS text.
+- Database schema upgrades and future-schema rejection.
+- Download path restrictions for CLI and MCP surfaces.
+- Cache refresh, sliding TTL, and cleanup date ranges.
+
+## Adding Tests
+
+When changing user-visible behavior, add tests at the surface where the behavior is observed. For example, add CLI tests for command output and exit behavior, MCP tests for tool return shapes, and lower-level tests for parser or database helpers.
+
+Keep fixtures local to a test file unless they are reused broadly. If a fixture touches home directories, database files, network calls, or environment variables, make sure it restores state automatically.