feat: production-grade SDK hardening (7-phase audit)

Phase 1 - Data loss & correctness:
- Default store_intermediate to True (prevents silent data loss)
- Smart defaults: build() no longer unconditionally injects doc_intel
- DataPackage.created_at uses server timestamp when available
- Replace silent exception swallowing with logged parse_warnings
- Add transient retry logic to pipeline wait() for HTTP 5xx
- Fix file_to_base64 filename parameter priority
- Fix PipelineResultResponse private fields with ConfigDict(extra=allow)

Phase 2 - Exception handling & async:
- Catch httpx.TransportError (subsumes Timeout + Connect)
- Add TransportError intermediate class in exception hierarchy
- Extract shared _parse_api_response to eliminate sync/async duplication
- Fix data.get("success") identity comparison
- Move Content-Type header from client init to post() only
- Fix AsyncPipeline.validate to async def
- Add NullHandler to logger; rename format -> fmt parameter
- Integrate request/response logging into base client

Phase 3 - DRY refactoring:
- Deduplicate enrichment models (import from chunking)
- Extract shared _inject_response_metadata / _build_common_request_body
- Consolidate retry constants into _constants.py
- Remove dead code (_looks_like_file_path, unused API_VERSION)
- Add cycle detection to topological sort

Phase 4 - Developer experience:
- Client-side validation for threshold, mode, dimension parameters
- from_yaml returns PipelineBuilder for method chaining
- add() resolves aliases and rejects duplicates
- Informative __repr__ on JobError / JobTimeoutError
- Add on_progress callback to wait() with proper Callable typing
- cancel() returns typed JobCancelResponse

Phase 5 - Model & type safety:
- Add RESUMABLE to JobStatus Literal
- Type pipeline_report as PipelineReport | None
- FileInput model_validator enforces at-least-one-of base64/url
- PipelineValidationError inherits from LatenceError
- Constrain Entity.score and Relation.confidence to [0.0, 1.0]
- Remove "chunking" from ServiceName Literal

Phase 6 - Test coverage (200 tests):
- New test_models.py: Pydantic parsing, constraints, all statuses
- New test_data_package.py: composition, merge, archive, warnings
- New test_builder_validation.py: smart defaults, strict, aliases, cycles

Phase 7 - Packaging:
- Add py.typed marker for mypy
- Pin httpx<1 and pydantic<3 upper bounds
- Secure API key: _api_key with @Property, masked __repr__

Critical fix found during final reflection:
- Fix execute() body construction to map configs by service NAME
  not positional INDEX (broken when validation auto-injects services)

Made-with: Cursor

Author: latenceainew

SDK_TUTORIAL.md

@@ -2,7 +2,7 @@
 
 This tutorial covers every feature of the Latence AI Python SDK: the **Data Intelligence Pipeline** for multi-stage document processing, **direct API access** to individual services, job management, credits, async usage, file handling, error handling, and configuration.
 
-> **Prerequisites**: Python 3.10+, a Latence API key from [app.latence.ai](https://app.latence.ai)
+> **Prerequisites**: Python 3.9+, a Latence API key from [app.latence.ai](https://app.latence.ai)
 
 ---
 
@@ -143,11 +143,12 @@ The `steps` dict lets you configure each stage. Keys are short aliases:
 | `knowledge_graph` or `ontology` | Relation Extraction |
 | `redaction` | PII Redaction |
 | `compression` | Text Compression |
-| `chunking` | Text Chunking |
 | `embedding` | Dense Embeddings |
 | `colbert` | ColBERT Embeddings |
 | `colpali` | ColPali Embeddings |
 
+> **Note**: Chunking is not available as a pipeline step. Use `client.experimental.chunking.chunk()` for standalone text chunking.
+
 ```python
 job = client.pipeline.run(
     files=["financial_report.pdf"],
@@ -222,11 +223,10 @@ builder.doc_intel(
     mode="default",              # "default" or "performance"
     output_format="markdown",    # "markdown", "json", "html", "xlsx"
     max_pages=None,              # limit pages processed
-    target_longest=None,         # image preprocessing dimension
-    use_layout_detection=True,
-    use_chart_recognition=True,
-    use_seal_recognition=False,
+    use_ocr_for_image_block=False,  # extract text from embedded images (+$0.25/1k pages)
 )
+# Layout detection, chart/seal recognition, and auto-rotate are pre-configured
+# for optimal pipeline results. For full control, use the direct API.
 
 # Entity Extraction
 builder.extraction(
@@ -247,52 +247,43 @@ builder.ontology(
     optimize_relations=True,           # refine relation labels (1.5x credits)
     predict_missing_relations=False,   # predict implicit links (2.5x credits)
     relation_threshold=0.5,
-    kg_output_format="custom",         # "custom", "property_graph", "rdf_turtle"
+    kg_output_format="custom",         # "custom", "property_graph", "rdf"
 )
 
 # Redaction
 builder.redaction(
-    mode="balanced",             # "balanced" or "strict"
+    mode="balanced",             # "balanced", "strict", "recall", "precision"
     threshold=0.3,
     redact=True,
     redaction_mode="mask",       # "mask" or "replace"
-    enable_refinement=False,
     chunk_size=1024,
 )
+# Full LLM refinement is always enabled in pipeline redaction for quality.
+# For manual refinement control, use the direct API.
 
 # Compression
 builder.compression(
     compression_rate=0.5,        # fraction of tokens to remove (0.0-1.0)
-    force_preserve_digit=False,
+    force_preserve_digit=True,   # preserve numeric tokens (default: True)
     force_tokens=None,           # tokens to always keep, e.g. ["API", "JSON"]
     apply_toon=False,            # TOON encoding (+$0.50/1M tokens)
-    chunk_size=512,
+    chunk_size=4096,             # max tokens per chunk (default: 4096)
     fallback_mode=True,
 )
 
-# Chunking
-builder.chunking(
-    strategy="hybrid",           # "character", "token", "semantic", "hybrid"
-    chunk_size=512,              # 64-8192
-    chunk_overlap=50,
-    min_chunk_size=64,
-    semantic_threshold=0.5,      # 0.1-0.95 (semantic/hybrid only)
-    semantic_window_size=3,      # 1-10 (semantic/hybrid only)
-)
-
-# Embedding
+# Embedding (experimental pipeline step)
 builder.embedding(
     dimension=512,               # 256, 512, 768, or 1024
     encoding_format="float",     # "float" or "base64"
 )
 
-# ColBERT
+# ColBERT (experimental pipeline step)
 builder.colbert(
     is_query=False,
     query_expansion=False,
 )
 
-# ColPali
+# ColPali (experimental pipeline step)
 builder.colpali(is_query=False)
 
 # Pipeline options
@@ -302,15 +293,36 @@ builder.strict()                 # disable auto-injection of services
 config = builder.build()
 ```
 
+> **Note**: Chunking is not available as a pipeline step -- `builder.chunking()` raises `NotImplementedError`. Use `client.experimental.chunking.chunk()` for standalone text chunking.
+
+### Pipeline execution model
+
+The pipeline worker executes services as a **directed acyclic graph (DAG)**, not a linear chain. Services that share the same parent can run concurrently:
+
+```
+                    ┌─── extraction ──── ontology
+                    │
+document_intelligence ─┼─── redaction
+                    │
+                    ├─── compression
+                    │
+                    ├─── embedding
+                    │
+                    ├─── colbert
+                    │
+                    └─── colpali
+```
+
+This means `extraction`, `redaction`, `compression`, and embedding services all run in parallel once `document_intelligence` completes. The SDK and worker automatically handle ordering -- you just declare which services you want.
+
 ---
 
 ## 5. Pipeline: PipelineConfig Object
 
 For maximum control, construct a `PipelineConfig` directly:
 
 ```python
-from latence import PipelineConfig
-from latence._models.pipeline import ServiceConfig
+from latence import PipelineConfig, ServiceConfig
 
 config = PipelineConfig(
     services=[
@@ -352,8 +364,6 @@ steps:
   document_intelligence:
     mode: performance
     output_format: markdown
-    use_layout_detection: true
-    use_chart_recognition: false
 
   extraction:
     label_mode: hybrid
@@ -419,6 +429,62 @@ job.cancel()
 pkg = job.data_package
 ```
 
+### Resumable jobs
+
+If a pipeline fails partway through, it may enter `RESUMABLE` status. Completed stages are checkpointed and only remaining stages re-execute on resume:
+
+```python
+try:
+    pkg = job.wait_for_completion()
+except JobError as e:
+    if e.is_resumable:
+        print(f"Job failed at a stage but is resumable: {e.message}")
+        pkg = job.resume().wait_for_completion()
+    else:
+        raise
+```
+
+### Intermediate results and report
+
+Access per-stage download URLs and the structured pipeline report while a job is running or after completion:
+
+```python
+# Per-stage download URLs (presigned B2 URLs to results.jsonl)
+stages = job.intermediate_results()
+for stage in stages:
+    print(f"{stage.service}: {stage.download_url}")
+
+# Structured pipeline report (dataset facts, per-stage metrics)
+report = job.report
+if report:
+    print(report)
+```
+
+### Validate before running
+
+Check a pipeline configuration without executing it:
+
+```python
+from latence import PipelineBuilder
+
+builder = PipelineBuilder().doc_intel().extraction().ontology()
+result = client.pipeline.validate(builder, files=["doc.pdf"])
+print(result.valid)       # True/False
+print(result.errors)      # list of errors
+print(result.warnings)    # list of warnings
+print(result.auto_injected)  # services auto-added
+```
+
+### Get available stages
+
+List the per-stage download links for a completed job:
+
+```python
+stages = client.pipeline.stages("pipe_abc123")
+for s in stages:
+    print(f"{s.service}: {s.download_url}")
+```
+
 ### Status values
 
 | Status | Meaning |
@@ -428,6 +494,7 @@ pkg = job.data_package
 | `COMPLETED` | Finished successfully |
 | `CACHED` | Results retrieved from cache |
 | `PULLED` | Results pulled from storage |
+| `RESUMABLE` | Failed partway through; call `job.resume()` to continue |
 | `FAILED` | Pipeline failed |
 | `CANCELLED` | Cancelled by user |
 
@@ -508,10 +575,10 @@ if pkg.compression:
 if pkg.chunking:
     print(pkg.chunking.summary.num_chunks)
     print(pkg.chunking.summary.strategy)       # "hybrid"
-    print(pkg.chunking.summary.avg_chunk_size)
+    print(pkg.chunking.summary.chunk_size)     # target chunk size parameter
 
     for chunk in pkg.chunking.chunks:
-        print(f"Chunk: {chunk[:80]}...")
+        print(f"Chunk: {chunk}")
 ```
 
 ### Enrichment section
@@ -525,18 +592,23 @@ if pkg.enrichment:
     for chunk in pkg.enrichment.chunks:
         print(chunk)
 
-    for feature_set in pkg.enrichment.features:
-        print(feature_set)
+    for name, data in pkg.enrichment.features.items():
+        print(f"{name}: {data}")
 ```
 
 ### Quality report
 
 ```python
 print(pkg.quality.total_cost_usd)
-print(pkg.quality.pipeline_name)
-print(pkg.quality.services_run)     # ["document_intelligence", "extraction", "ontology"]
-print(pkg.quality.total_stages)
-print(pkg.quality.execution_summary)
+print(pkg.quality.total_processing_time_ms)
+
+for stage in pkg.quality.stages:
+    print(f"{stage.service}: {stage.status} ({stage.processing_time_ms}ms, ${stage.credits_used})")
+
+# Confidence scores
+print(pkg.quality.confidence.entity_avg_confidence)
+print(pkg.quality.confidence.graph_completeness)
+print(pkg.quality.confidence.ocr_quality)
 ```
 
 ### Download as ZIP archive
@@ -548,28 +620,37 @@ print(f"Saved to {path}")
 
 Archive structure:
 ```
-results/
+{pipeline_name}/
   README.md
   document.md
   pages/
     page_001.md
     page_002.md
   entities.json
   knowledge_graph.json
-  redaction.json
-  compression.json
-  summary.json
+  redaction.json          (if redaction ran)
+  compression.json        (if compression ran)
+  chunking.json           (if chunking ran)
+  enrichment.json         (if enrichment ran)
+  quality_report.json
+  metadata.json
 ```
 
 ### Merge into flat dict
 
 ```python
 merged = pkg.merge()
 # {
+#   "id": "pipe_xxx",
+#   "name": "My Pipeline",
+#   "status": "COMPLETED",
+#   "created_at": "2025-...",
 #   "documents": [{"filename": "doc.pdf", "markdown": "...", "entities": [...], ...}],
-#   "stats": {"documents": 1, "pages": 5, "entities": {...}, ...},
-#   "meta": {"pipeline_name": "...", "services": [...], ...},
+#   "summary": {"documents": 1, "pages": 5, "entities": {...}, "relations": {...}, ...},
 # }
+
+# Save merged output directly to a JSON file:
+pkg.merge(save_to="./results.json")
 ```
 
 ---
@@ -827,7 +908,7 @@ result = red.detect_pii(
 result = red.detect_pii(
     text="...",
     config={
-        "mode": "balanced",              # "balanced" | "strict"
+        "mode": "balanced",              # "balanced" | "strict" | "recall" | "precision"
         "threshold": 0.3,               # confidence threshold (0.0-1.0)
         "redact": True,                 # detect only vs. detect and redact
         "redaction_mode": "mask",        # "mask" | "replace"
@@ -879,7 +960,7 @@ result = ont.build_graph(
         "resolve_entities": True,              # merge duplicates (2.0x credits)
         "optimize_relations": True,            # refine labels with LLM (1.5x credits)
         "predict_missing_relations": True,     # predict implicit links (2.5x credits)
-        "kg_output_format": "custom",          # "custom" | "rdf_turtle" | "property_graph"
+        "kg_output_format": "custom",          # "custom" | "rdf" | "property_graph"
         "relation_threshold": 0.6,
         "symmetric": True,
         "generate_knowledge_graph": True,
@@ -901,14 +982,14 @@ result = ont.build_graph(
 )
 ```
 
-### RDF/Turtle format
+### RDF format
 
 ```python
 result = ont.build_graph(
     text="...",
     entities=[...],
     config={
-        "kg_output_format": "rdf_turtle",
+        "kg_output_format": "rdf",
         "namespace_uri": "http://example.org/ontology#",
     },
 )

pyproject.toml

@@ -24,8 +24,8 @@ classifiers = [
     "Typing :: Typed",
 ]
 dependencies = [
-    "httpx[http2]>=0.27.0",
-    "pydantic>=2.0.0",
+    "httpx[http2]>=0.27.0,<1",
+    "pydantic>=2.0.0,<3",
 ]
 
 [project.optional-dependencies]

src/latence/__init__.py

@@ -84,6 +84,7 @@
     NotFoundError,
     RateLimitError,
     ServerError,
+    TransportError,
     ValidationError,
 )
 
@@ -130,6 +131,7 @@
     PipelineReport,
     PipelineResultResponse,
     PipelineSubmitResponse,
+    ServiceConfig,
     StageDownload,
     StageStatus,
 )
@@ -151,6 +153,7 @@
     "ValidationError",
     "RateLimitError",
     "ServerError",
+    "TransportError",
     "APIConnectionError",
     "APITimeoutError",
     "JobError",
@@ -160,7 +163,6 @@
     "Job",
     "AsyncJob",
     "DataPackage",
-    "PipelineBuilder",
     # Common models
     "Entity",
     "KnowledgeGraph",
@@ -191,6 +193,7 @@
     "PipelineReport",
     "PipelineResultResponse",
     "PipelineSubmitResponse",
+    "ServiceConfig",
     "StageDownload",
     "StageStatus",
 ]