Skip to content

Commit 2279f3f

Browse files
authored
feat(sources): log the effective content-core extraction engine (#1125)
* feat(sources): log the effective content-core extraction engine The source-processing graph now emits an INFO line naming the resolved url_engine / document_engine / docling_ocr right before extraction. content-core only logs its own engine dispatch at DEBUG, so operators had no way to confirm which engine actually ran for a given source (e.g. whether a persisted Crawl4AI selection took effect or the request silently fell back to the 'auto' chain). Falls back to 'auto' in the log when no override is set, matching content-core's default. * docs(changelog): note the extraction-engine logging (#1125)
1 parent 906ad16 commit 2279f3f

2 files changed

Lines changed: 17 additions & 0 deletions

File tree

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
1111
- **Opt-in heavy extraction runtimes.** Docling (layout parsing + OCR + image sources) and local Crawl4AI (JavaScript rendering) are now **opt-in**, installed automatically on first container startup when enabled — instead of being bundled into (or missing from) every image. Set `OPEN_NOTEBOOK_ENABLE_DOCLING=true` and/or `OPEN_NOTEBOOK_ENABLE_CRAWL4AI=true`; the downloads are cached on the `/app/data` volume so only the first boot is slow, and a failed install degrades gracefully (the app still starts, the engine is reported unavailable) with loud logs. A new `GET /api/capabilities` probe reports what's actually installed, and Settings → Content Processing disables the Docling engine, the Crawl4AI engine and the OCR toggle (with an env-var hint) until their runtime is available — so the UI never advertises an engine that isn't there. This keeps the default image lean (no Chromium, no multi-GB ML stack) while making both runtimes available on demand. A remote Crawl4AI server via `CRAWL4AI_API_URL` needs no local install. New hint strings added across all 14 locales; recorded as ADR-007 (#1122)
1212
- **OCR toggle** in Settings → Content Processing: a new "Enable OCR" checkbox controls whether the Docling engine runs OCR on scanned PDFs and images. It's on by default (matching content-core's behavior); turn it off to speed up processing of text-native documents. The setting is passed to content-core's `docling_ocr` config, and the label/help are translated across all 14 locales. OCR only runs through Docling, so the toggle takes effect once Docling is enabled (see #1122) (#1104)
1313
- **Crawl4AI** is now selectable as a URL processing engine in Settings → Content Processing, alongside Firecrawl, Jina and Simple. It renders JavaScript-heavy pages locally with no API key, or offloads to a Crawl4AI server when `CRAWL4AI_API_URL` is set. Local Crawl4AI (and its Chromium browser) is an opt-in runtime installed on first startup when `OPEN_NOTEBOOK_ENABLE_CRAWL4AI=true` (see #1122) — the default image stays lean. As part of this, the persisted document/URL engine choices now actually take effect: the source-processing graph reads the saved Content Settings and passes them to content-core (previously it always ran with hard-coded `auto` engines and silently ignored the selection). New engine label added across all 14 locales (#432)
14+
- Source extraction now logs the **effective content-core engine** at INFO (`url_engine` / `document_engine` / `docling_ocr`) right before it runs. content-core only logs its own engine dispatch at DEBUG, so operators previously had no way to confirm which engine actually processed a source — e.g. whether a persisted Crawl4AI/Docling selection took effect or the request silently fell back to the `auto` chain (#1125)
1415

1516
### Changed
1617
- Upgraded the content extraction dependency from content-core 1.14.x to 2.x (2.0.4). The source-processing graph was adapted to content-core's new keyword-only `extract_content()` API: engine/model overrides now travel through a `ContentCoreConfig` object instead of the input dict, and the extraction result (`ExtractionOutput`) no longer echoes the source `url`/`file_path` back, so those are carried from the request state into the saved source asset. Because content-core 2.x no longer deletes the uploaded file after extraction, the graph now honors the `delete_source` flag itself. Transitively this replaces the AGPL-licensed PyMuPDF with MIT-licensed pdfplumber for PDF extraction and drops moviepy in favor of direct ffmpeg calls (which fixes audio extraction from MP3 files carrying chapter metadata). No user-facing configuration changes in this step — document and URL engines stay on their `auto` defaults; new engine/OCR options are tracked separately under #939 (#1103)

open_notebook/graphs/source.py

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,22 @@ async def content_process(state: SourceState) -> dict:
9797

9898
config = ContentCoreConfig(**config_kwargs) if config_kwargs else None
9999

100+
# Log the effective extraction engines so operators can confirm which engine
101+
# actually ran (content-core logs its own dispatch only at DEBUG). Absent
102+
# overrides fall back to content-core's "auto".
103+
if content_state.get("url"):
104+
target = "url"
105+
elif content_state.get("file_path"):
106+
target = "document"
107+
else:
108+
target = "content"
109+
logger.info(
110+
f"Extracting {target} via content-core "
111+
f"(url_engine={config_kwargs.get('url_engine', 'auto')}, "
112+
f"document_engine={config_kwargs.get('document_engine', 'auto')}, "
113+
f"docling_ocr={config_kwargs.get('docling_ocr', 'auto')})"
114+
)
115+
100116
processed = await extract_content(
101117
url=content_state.get("url"),
102118
file_path=content_state.get("file_path"),

0 commit comments

Comments
 (0)