Production hardening: /v1 API on real Ray Serve, standalone client, plugins, Docker, observability (v0.6.0) - #46
Merged
Merged
Conversation
…ostic core Foundational hardening ahead of the Ray Serve migration. Three phases, suite green at 105 tests (was 83). Phase 0 - CI truth & packaging - Fix coverage target (--cov=pixano_inference) and add a CPU-torch CI job so model tests run instead of silently skipping; add a framework-free-core CI job. - Move the sam-2 git dependency out of the published [sam2] extra into the dev dependency-group so the wheel carries no direct/VCS reference (PyPI-publishable). - Pin ray[serve] >=2.53,<3.0; add license metadata; point project.urls at GitHub; fix the pytest-asyncio dependency and add [tool.pytest.ini_options]. - Remove empty pre-Ray test dirs and stale custom_yoloe references; exclude build/venv/site dirs from the license-header check. Phase 1 - Security baseline - Optional API-key auth (X-API-Key / Bearer, constant-time) on all inference and tracking-job routes, configured via ServerSettings (PIXANO_INFERENCE_* env). - MediaPolicy for media ingestion: http/https only, private/loopback/metadata-IP blocking, connect/read timeouts, streamed size caps, redirect re-validation, and media-root path containment. Closes the SSRF and arbitrary-file-read surface. - Body-size limit + optional CORS middleware; sanitize 500 responses; default bind host 0.0.0.0 -> 127.0.0.1. Phase 1.5 - Framework-agnostic core - New frameworks/ adapter package (torch/jax/tensorflow/mlx) behind a lazy registry; the core depends on numpy only. Torch tensor conversions moved off the NDArray wire type and utils/vector.py removed. Torch adapter gains an Apple-Silicon MPS path. - Add jax/tensorflow/mlx optional extras and package-detection helpers. - A guardrail test asserts the core plus all four adapter modules import with no ML framework installed.
Replace the plain @ray.remote actor-per-model design with real Ray Serve deployments, making autoscaling, replicas, batching, and crash-restarts functional and adding a graceful lifecycle and truthful health checks. Serve deployments (ray/deployment.py) - ModelReplica is a @serve.deployment; build_model_app() sets a fixed replica count when min==max, otherwise an autoscaling_config, plus ray_actor_options, max_ongoing_requests, and health_check_period_s. The model class is bound by value so no driver registry state is needed in the replica. - Model access is serialised on a per-replica single worker thread (stateful models are never entered concurrently); batching is opt-in via @serve.batch when max_batch_size > 1. - __del__ calls model.unload(), fixing the GPU leak on undeploy; check_health added. DeploymentManager (ray/app.py) - Uses serve.run / serve.delete / serve.get_app_handle / serve.status. Pre-flight resource check fails fast instead of hanging 300s on GPU oversubscription. readiness() reports 503 unless every model is RUNNING. Async tracking-job store over DeploymentResponse (no event-loop ray.get) with TTL + max-jobs eviction. - A FastAPI lifespan owns Ray init + serve.start (HTTP proxy disabled) + strict startup deploys + drain (serve.shutdown/ray.shutdown), so SIGTERM shuts down gracefully. Request path & health (ray/routes) - _run_inference awaits handle.predict.remote(...) with asyncio.wait_for -> 504 on timeout (per-capability defaults, per-model override); 500s no longer leak exception text. - /ready is real (503 until all models RUNNING); /app/settings reports live cluster status. Config & CLI - min_replicas 0->1, max_batch_size 8->1; add max_ongoing_requests, health_check_period_s, timeout_s, strict_startup, ray_address, ray_namespace. models/base gains predict_batch; the CLI gains --strict-startup. Tests - Rewrite tests/test_ray_app.py off the ray.get/ray.wait monkeypatch (awaitable Serve-style handles; async job-manager unit tests; readiness tests). Add tests/integration that boot a real local Ray + Serve runtime with a numpy-only stub model and verify deploy -> RUNNING -> predict -> undeploy, fixed replicas, batching, pre-flight GPU rejection, and double-deploy. CI gains a separate integration job; the fast job runs -m "not integration".
Introduce a clean, versioned HTTP contract and delete the unversioned routes. The whole
/v1 wire contract is camelCase (Python code stays snake_case via populate_by_name), typed,
and documented; long-running tracking runs as bounded async jobs.
/v1 API (src/pixano_inference/api/v1/)
- router.py mounts everything under /v1 with a top-level /health alias; auth guards
inference/jobs/admin while health/ready/info stay open for probes.
- inference.py: segmentation/detection/vlm/ner/tracking (+ binary multipart variants),
typed response_model so responses are validated and camelCased.
- jobs.py: submit (202) / poll / cancel tracking jobs.
- admin.py: runtime deploy/undeploy (serve.run/serve.delete via anyio.to_thread), model
listing with live Serve status, and per-model stats.
- errors.py: a consistent {"error": {code, message, requestId}} envelope; unhandled errors
never leak their text.
- schemas.py: the frontend's nested keyframes[].prompts request shape, mapped onto the flat
internal TrackingKeyframe.
JobManager (src/pixano_inference/jobs.py)
- Extracted from DeploymentManager into a generic, bounded (TTL + max-jobs) manager over
Serve DeploymentResponses; the manager now delegates to it.
Wire contract
- CamelModel base applied to the core I/O types and BaseRequest/BaseResponse.
- NDArray serializes as {shape, dtype, data(base64)} instead of a JSON float list, so SAM2
embeddings/logits are compact.
- NER is now servable (route + schemas + capability); models/llm.py and the unused
APIRequest are removed.
Contract lockstep
- Commit docs/openapi.json + scripts/gen_openapi.py; CI (--check) keeps it in sync so the
Pixano frontend can regenerate TypeScript types.
Tests
- tests/api/test_v1.py (camelCase, NDArray binary round-trip, NER, error envelope, jobs),
rewritten tests/test_ray_app.py (manager/JobManager/service), and
tests/integration/test_v1_e2e.py (real Serve behind the app: deploy -> /v1/detection ->
camelCase -> /v1/models RUNNING). 107 tests pass.
…, auth)
Replace the old client (a Settings subclass mixing requests + httpx + fastapi, targeting
the removed unversioned routes) with a clean /v1 client.
client.py
- httpx-only; drops the fastapi and requests dependencies and the Settings inheritance.
- PixanoInferenceClient (async) + SyncPixanoInferenceClient (sync twin) share a base with a
pooled transport, optional X-API-Key auth, retry-with-backoff on 502/503/504 and
connect/timeout errors, and per-call timeout overrides (long default for tracking).
- Bodies serialize as camelCase JSON (model_dump(mode="json", by_alias=True)); errors raise
PixanoInferenceError carrying the {code, message, requestId} envelope.
- Typed methods for every /v1 route: segmentation/detection/vlm/ner/tracking,
submit/get/cancel/wait_for_job, list/deploy/undeploy models, and info/ready/health
(ready does not raise on 503). A transport= hook enables in-process ASGI testing.
Schemas
- Move the client-facing /v1 wire schemas (tracking request, job/model envelopes) to
schemas/v1.py so the client never imports a web framework; api/v1/schemas.py re-exports.
- Delete the vestigial settings.py.
Tests
- Rewrite tests/test_client.py on pytest_httpx (auth header, camelCase body,
error->exception, 503 retry, job lifecycle, sync twin) and add
tests/integration/test_client_e2e.py (real client driving the live app over ASGI:
health/ready/detection/list_models). 103 tests pass.
Make extension a packaging concern instead of a path concern. A custom model is now an independent, pip/uv-installable package that advertises itself via a pixano_inference.models entry point; the server discovers it automatically at startup, and because it is installed it is importable in every Ray Serve worker. This removes --module-path entirely and lays the groundwork for a shared "model store" (the built-in SOTA models can ship the same way). Discovery (plugins.py) - load_plugin_models() imports every pixano_inference.models entry point (running their @register_model decorators) and skips+logs a broken plugin rather than aborting startup. - ensure_models_loaded() registers built-in backends and plugins idempotently; wired into config resolution and app creation. CLI - Remove --module-path (an editable install gives live iteration, discovery, and worker-importability). Config resolution warns when a model class is defined in __main__. Examples - examples/numpy_detector: a real installable, framework-free (numpy-only) DetectionModel package with its own pyproject + entry point + README — the canonical extension example. Added as an editable dev dependency so its discovery is exercised by the tests. - examples/yolo: converted to the same plugin-package layout (src/pixano_yolo + pyproject + entry point), config references the model by name, client script updated, AGPL note added. Docs & tests - Rewrote docs/ray_serve/custom_models.md around packages and the model-store vision. - tests/test_plugins.py (real discovery + broken-plugin skip) and an integration test that deploys NumpyDetector through real Ray Serve with no torch. CI installs the example plugin in the fast, integration, and framework-free jobs. 108 tests pass.
Move the SAM2 models out of the core package and into packages/pixano-inference-sam, a standalone plugin distributed and discovered exactly like any third-party model. This proves the built-in SOTA models are just first-party plugins and is the first step of the "model store". Plugin (packages/pixano-inference-sam) - pixano_inference_sam: image.py, video.py, params.py, _prompts.py (the SAM point/box helpers, previously in impls/_helpers), with a pyproject.toml, a pixano_inference.models entry point, and a README. Importing the package registers Sam2ImageModel / Sam2VideoModel; the upstream sam-2 library is only needed at load time (load_model asserts it). Core removals - Delete impls/sam2, impls/sam3, configs/sam2; drop the sam2/sam3 guards from impls/__init__, is_/assert_sam3_installed, the SAM param export from configs/__init__, and the SAM prompt helpers from impls/_helpers. Packaging - Replace the [sam2]/[sam3] extras with [sam] = ["pixano-inference-sam"], so `pip install pixano-inference[sam]` installs the plugin + torch (the git-only sam-2 library remains a documented one-line install). The core wheel now bundles no SAM code and carries no direct references. Tests/CI - SAM tests stay in the core suite but import the plugin (importorskip); test_configs imports SAM params from the plugin. The plugin is wired into the dev env and the fast CI job. 108 tests pass.
Add a production-oriented container setup and validate it end-to-end. A multi-lens adversarial review of the (locally unbuildable) GPU path drove the hardening below. Image (Dockerfile) - Multi-stage on python:3.11-slim with CUDA torch wheels (no CUDA base image; GPU via the host driver + NVIDIA Container Toolkit). Non-root uid 1000, HF_HOME on the /data volume, HEALTHCHECK on /health. Build args TORCH_INDEX_URL / PIXANO_EXTRAS / INSTALL_SAM / INSTALL_EXAMPLE produce GPU, CPU, and framework-free variants; the SAM2 plugin + a pinned sam-2 commit are bundled by default. - Review fixes: install torchvision from the same index as torch (was PyPI -> CPU/ABI mismatch); add gcc/g++ so torch.compile works; 600s healthcheck start period for cold model loads. Compose - GPU reservation, shm_size 8gb (Ray object store), weights volume, init: true (reap Ray children), restart: unless-stopped, and a stop_grace_period matched to a bounded uvicorn graceful drain. Port bound to 127.0.0.1 by default; exposing off-host requires setting PIXANO_INFERENCE_API_KEYS (secure default). Server - Bound uvicorn timeout_graceful_shutdown (new RayServeConfig.graceful_shutdown_s) so the drain + serve/ray teardown finish before SIGKILL. Also: docker/models.py (SAM, compile=False) + docker/models.numpy.py (framework-free) example configs, docs/deployment/docker.md, a .dockerignore that excludes .env/secrets/media, and a CI docker_smoke job that builds the framework-free image and asserts /v1/ready. The minimal image was built and run locally: non-root, Serve boots, the numpy plugin deploys, and a real detection request returns the correct box.
…lugin Add a unified `embedding` capability that turns an image or text into a vector in one shared CLIP space (for semantic / text-to-image search, dedup, clustering) — same architecture as every other capability: the contract lives in core, the model ships as an entry-point plugin. Core (framework-free): - models/embedding.py: EmbeddingModel (capability_name="embedding"), EmbeddingInput (image XOR text, single or list, normalize), EmbeddingOutput (embeddings: NDArrayFloat [N, dim] + dim); registered in HTTP_CAPABILITY_BASES. - schemas/inference.py: EmbeddingRequest/EmbeddingResponse (camelCase wire, compact binary NDArray). - api/v1/inference.py: POST /v1/inference/embedding (+ /binary multipart). - ray/app.py: 60s embedding timeout. client.py: embedding() on both clients. - Regenerate docs/openapi.json. - Fix api/v1/errors.py to jsonable_encode error detail/errors so a pydantic ValidationError carrying a ValueError no longer 500s the error envelope. Plugin packages/pixano-inference-clip (open_clip): - OpenClipEmbeddingModel embeds image or text into the same space; open_clip imported lazily at load_model so discovery stays light. Default checkpoint MobileCLIP2-S2 / dfndr2b — recent, performant, CPU-friendly; any open_clip spec works. - Wired into core [clip] extra + dev group + editable uv source. Tests + CI: - v1 (text/image share the binary wire; image XOR text -> 422), client binary parse, plugin discovery, and a real MobileCLIP2 deploy/embed integration test (image + text land in one 512-d space). Clip plugin installed in the fast and integration CI jobs.
Third-party apps that only call a Pixano Inference server no longer need the
server stack (ray[serve], fastapi, uvicorn, torch). The client + wire schemas
ship as a separate, lightweight distribution.
New package packages/pixano-inference-client (import pixano_inference_client):
- Holds the whole wire contract — CamelModel/BaseRequest/BaseResponse, NDArray,
CompressedRLE, ModelInfo, the per-capability I/O types, the *Request/*Response
models, the v1 admin/job types, is_url — plus both client classes.
- Hard deps are just httpx, pydantic, numpy, typing_extensions. pycocotools and
Pillow are lazy-imported in rle.py and gated behind an optional [masks] extra,
so parsing a normal response needs neither.
Core pixano_inference is now thin re-export shims at every legacy path
(schemas/{base,nd_array,rle,models,inference,v1}, models/<task> keep only the
server *Model base and re-export their I/O, client), so the whole repo and the
sam/clip plugins keep their imports unchanged: from pixano_inference.client
import PixanoInferenceClient and from pixano_inference_client import
PixanoInferenceClient resolve to the same classes. The models<->schemas import
cycle is gone, so schemas/__init__ drops its lazy __getattr__.
Packaging:
- Root depends on pixano-inference-client[masks] (dropped the now-transitive
httpx) and wires it as an editable uv source; the Dockerfile and every CI job
install it before the core (hard dep, not yet on PyPI).
- publish.yml gains a deploy_pypi_client job (the root job needs it, so the
client is published first).
- New client_standalone CI job installs only the client and asserts the server
stack is absent; the package's own tests assert importing it pulls in no
ray/fastapi/uvicorn/torch.
A client-only install is ~46 MB (numpy/pydantic/pillow/pycocotools) versus the
full server (ray + torch, hundreds of MB to GBs).
Add an observability layer and a couple of robustness fixes. New src/pixano_inference/observability.py: - RequestContextMiddleware (pure ASGI): tag every request with an X-Request-ID (echoed from the client or generated) exposed on request.state, the response header, and a ContextVar so it reaches endpoints and log records. - PrometheusMiddleware: request count, latency histogram, and in-flight gauge, labelled by method and matched route template (bounded cardinality). Metric objects are module-level singletons so rebuilding the app never double-registers. - render_metrics() + a /metrics (and /v1/metrics) unauthenticated scrape route. - configure_logging(): dictConfig with a request-id-aware plain or JSON formatter. Wiring: install_observability_middleware() in ray/app.py (request id outermost); the error envelope now prefers request.state.request_id; main.py configures logging from ServerSettings.log_level / log_json. Robustness: - should_compile(device, requested) in impls/_helpers: honour an explicit compile flag else auto-detect (GPU only). Applied to the transformers grounding_dino and vlm backends, which were calling torch.compile unconditionally (a slow no-win or outright failure on CPU). - Add prometheus-client as a core dependency. - scripts/load_test.py: an async httpx load generator reporting throughput and latency percentiles, for a GET probe or a POST inference route. Tests: tests/test_observability.py covers request-id generation/echo/propagation, the metrics endpoint and route-template labelling, the logging filter, and should_compile. The /metrics routes are include_in_schema=False so the committed OpenAPI schema is unchanged.
The standalone client inherited the server's `numpy < 2.0.0` pin, but its NDArray wire code (from_numpy/to_numpy, base64 raw-bytes encoding) is fully numpy-2 compatible, and downstream consumers — the Pixano app locks numpy 2.x — would be forced into a needless downgrade. Relax the client to `numpy >= 1.26.0, < 3.0.0`; the server keeps its own `< 2.0.0` pin, so this repo's resolved environment is unchanged (still 1.26.4). Verified in a clean venv: the client wheel installed alongside numpy 2.5.1 passes its test suite and an exact NDArray round-trip.
ModelConfig resolved model_params in a before-validator that consulted
ModelParamsRegistry, but plugin loading (ensure_models_loaded) only happened
later in model_post_init. A plugin model referenced from a config file with no
explicit model_params therefore silently lost its registered param defaults,
and the Serve replica failed at load time with KeyError('path') — e.g. the
bundled CLIP embedding model deployed via
`pixano-inference --config models.py` with default params.
Call ensure_models_loaded() (idempotent) in the params validator so both
resolution steps are self-sufficient regardless of construction order. Found
by validating the real config-file deploy path end-to-end; regression test
runs the cold path in a fresh interpreter.
Summarize the /v1 API, real Ray Serve backend, security baseline, framework-agnostic core, plugin packages, standalone client, Docker deployment, and observability work, with the breaking changes called out for upgraders.
The pre-commit CI job on this branch never actually ran (it always died at `pip install .`, which cannot resolve the not-yet-published pixano-inference-client), so lint/type/format debt and a few real CI-config problems went undetected. Fix them all: - pre-commit (lint_and_format): drop `pip install .` — the hooks run in isolated environments and do not need the package installed. Exclude the example deployment scripts in deploy/ from ruff (force-exclude) and mypy, add types-requests to the mypy hook, and .prettierignore the hand-managed CLAUDE.md. Reformat the Phase 5/6 docs + docker-compose with Prettier, and the docker/models.py import order with ruff. - OpenAPI check: move it into its own job that runs against the locked environment (uv sync), so it is deterministic — a fresh pip resolve pulls a newer FastAPI that emits a slightly different schema than the committed one. - Ray Serve integration + backend: install torch AND torchvision from the same CPU index so their compiled ops stay ABI-coherent (open_clip pulled a PyPI torchvision against the CPU-index torch -> "operator torchvision::nms does not exist"). - Docker: `uv pip install .` resolves the client via [tool.uv.sources] but installs it editable (a path into /build), which the runtime stage's venv copy loses; reinstall it as a real package so it survives the copy. - docs / publish: install the client editable before `.[docs]` (hard core dep, not yet on PyPI).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Production hardening → v0.6.0
Rebuilds the serving stack on real Ray Serve behind a versioned, camelCase
/v1API,and adds a security baseline, a framework-agnostic core, installable plugin packages, a
standalone client distribution, single-node Docker deployment, and observability. Validated
end-to-end against the Pixano app (segmentation, tracking, and CLIP embeddings for explorer
search).
12 commits, ~138 files. See
CHANGELOG.mdfor the full 0.6.0 entry.Why
Commit
1ab6f87left the Ray Serve migration incomplete: the code carried Ray Serve naming and afull autoscaling/batching config surface but actually ran plain
@ray.remoteactors behind anin-process app — so the advertised scaling was dead config, and security, lifecycle,
observability, and packaging were unaddressed. An audit mapped the gaps; this closes them.
/v1API — versioned, typed, camelCase wire; error envelope{"error": {code, message, requestId}}; 500s sanitized. Old unversioned/snake_case routes removed.NDArraywire ({shape, dtype, data}, base64) instead of JSON float lists.pixano-inference-clientdistribution(httpx/pydantic/numpy only).
from pixano_inference.client import PixanoInferenceClientstillworks with the full install.
pixano_inference.models);--module-pathremoved.pixano-inference-sam—pip install pixano-inference[sam].127.0.0.1by default; external exposure requires API keys.What's in it
lifecycle, pre-flight resource checks, runtime deploy/undeploy, async
JobManager, 504 timeouts.embedding(image+text shared CLIP space, bundledpixano-inference-clip/ MobileCLIP2) and NER.
MediaPolicy) + media-rootcontainment.
framework-free variants.
X-Request-IDmiddleware, Prometheus/v1/metrics, request-id-aware logging.docs/openapi.json(+ generator), CI-checked, for generated frontend types.pixano-inference-client,pixano-inference-sam,pixano-inference-clip.Testing
Docker build-and-smoke jobs added to CI.
sam-2stays in the dev group + image.segmentation, tracking, and embedding search against the running server.
Notes for reviewers
uv.lock.src/pixano_inference/ray/(serving core) →api/v1/(contract) →packages/pixano-inference-client/(the client boundary) →security.py/utils/media_security.py→Dockerfile/observability.py.pixano-inference-clientPyPI project + trusted publisher before the first tagged release(the publish workflow is already wired for it).