Migrate to Ray Serve and capability-based inference API (#45)

Author: jrabary

.env_example

@@ -11,3 +11,7 @@ dist/
 docs/build/*
 docs/source/generated/*
 .env
+site/
+CLAUDE.md
+celery_logs.log
+poc/

.gitignore

@@ -32,7 +32,11 @@
 
 ## Context
 
-This library aims to provide a common ecosystem to launch inference for various Artificial Intelligence tasks from different providers (Open-AI, transformers, sam2, ...). It has first been implemented to work in par with the [Pixano](https://pixano.github.io/pixano/latest/) AI-powered annotation tool.
+This library provides a Ray Serve-based inference server for multimodal AI
+tasks. It was first built to support the
+[Pixano](https://pixano.github.io/pixano/latest/) AI-powered annotation tool
+and exposes typed deployment configs, a Python client, and a REST API for
+running deployed models.
 
 ## Installation
 

README.md

@@ -0,0 +1,31 @@
+# =================================
+# Copyright: CEA-LIST/DIASI/SIALV
+# Author : pixano@cea.fr
+# License: CECILL-C
+# =================================
+
+"""SAM2 deployment configuration for Pixano Inference.
+
+Usage:
+    pixano-inference --config deploy/sam2_example.py
+"""
+
+from pixano_inference.configs import DeploymentConfig, ModelConfig, Sam2ImageParams, Sam2VideoParams
+from pixano_inference.impls.sam2.image import Sam2ImageModel
+from pixano_inference.impls.sam2.video import Sam2VideoModel
+
+
+models = [
+    ModelConfig(
+        name="sam2-image",
+        model_class=Sam2ImageModel,
+        model_params=Sam2ImageParams(path="facebook/sam2.1-hiera-tiny", torch_dtype="float32"),
+        deployment=DeploymentConfig(num_gpus=0, min_replicas=0, max_replicas=1, max_batch_size=8),
+    ),
+    ModelConfig(
+        name="sam2-video",
+        model_class=Sam2VideoModel,
+        model_params=Sam2VideoParams(path="facebook/sam2.1-hiera-tiny", torch_dtype="float32"),
+        deployment=DeploymentConfig(num_gpus=0, min_replicas=0, max_replicas=1, max_batch_size=1),
+    ),
+]

deploy/sam2_example.py

@@ -0,0 +1,174 @@
+<!---
+# =================================
+# Copyright: CEA-LIST/DIASI/SIALV
+# Author : pixano@cea.fr
+# License: CECILL-C
+# =================================
+--->
+
+# Pixano Inference HTTP API
+
+This document describes the HTTP API exposed by the Ray Serve-based Pixano
+Inference server.
+
+**Base URL:** `http://<host>:<port>` with default `http://127.0.0.1:7463`
+
+Start the server with a Python config file:
+
+```bash
+pixano-inference --config models.py
+```
+
+## Overview
+
+- Models are loaded at startup from a Python `.py` config file passed to `--config`.
+- All inference routes are synchronous `POST` endpoints.
+- There are no runtime HTTP endpoints for deploying or undeploying models.
+- Every inference request includes a `model` field that must match a deployed model name.
+- Endpoint families are capability-based: segmentation, detection, tracking, and VLM.
+
+## Service endpoints
+
+| Method | Path             | Purpose                              |
+| ------ | ---------------- | ------------------------------------ |
+| `GET`  | `/`              | Basic API metadata and docs link     |
+| `GET`  | `/health`        | Liveness probe                       |
+| `GET`  | `/ready`         | Readiness summary                    |
+| `GET`  | `/app/settings/` | Server settings and resource summary |
+| `GET`  | `/app/models/`   | List deployed models                 |
+
+### `GET /app/settings/`
+
+Example response:
+
+```json
+{
+  "app_name": "Pixano Inference",
+  "app_version": "0.6.0",
+  "app_description": "Pixano Inference API powered by Ray Serve",
+  "num_cpus": 8,
+  "num_gpus": 2,
+  "num_nodes": 1,
+  "gpus_used": 1.0,
+  "gpu_to_model": {},
+  "models": ["sam2-image"],
+  "models_to_capability": {
+    "sam2-image": "segmentation"
+  }
+}
+```
+
+### `GET /app/models/`
+
+Returns a list of `ModelInfo` objects:
+
+```json
+[
+  {
+    "name": "sam2-image",
+    "capability": "segmentation",
+    "model_path": "facebook/sam2-hiera-base-plus",
+    "model_class": "Sam2ImageModel"
+  }
+]
+```
+
+## Inference endpoints
+
+| Method | Path                       | Request schema        | Response schema        | Python client helper    |
+| ------ | -------------------------- | --------------------- | ---------------------- | ----------------------- |
+| `POST` | `/inference/segmentation/` | `SegmentationRequest` | `SegmentationResponse` | `client.segmentation()` |
+| `POST` | `/inference/detection/`    | `DetectionRequest`    | `DetectionResponse`    | `client.detection()`    |
+| `POST` | `/inference/tracking/`     | `TrackingRequest`     | `TrackingResponse`     | `client.tracking()`     |
+| `POST` | `/inference/vlm/`          | `VLMRequest`          | `VLMResponse`          | `client.vlm()`          |
+
+If a model exists but does not support the endpoint capability, the server
+returns `400`.
+
+The request and response models are available from `pixano_inference.schemas`.
+
+### Example: segmentation
+
+```json
+{
+  "model": "sam2-image",
+  "image": "data:image/png;base64,...",
+  "points": [[[200, 175]]],
+  "labels": [[1]]
+}
+```
+
+### Example: detection
+
+```json
+{
+  "model": "grounding-dino",
+  "image": "http://images.cocodataset.org/val2017/000000039769.jpg",
+  "classes": ["cat", "remote control"],
+  "box_threshold": 0.3,
+  "text_threshold": 0.2
+}
+```
+
+## Response envelope
+
+All inference endpoints return the same top-level envelope:
+
+```json
+{
+  "id": "ray-sam2-image-1739000000000",
+  "status": "SUCCESS",
+  "timestamp": "2026-01-01T12:00:00",
+  "processing_time": 0.234,
+  "metadata": {
+    "model_name": "sam2-image",
+    "capability": "segmentation",
+    "model_class": "Sam2ImageModel"
+  },
+  "data": {}
+}
+```
+
+| Field             | Description                                                |
+| ----------------- | ---------------------------------------------------------- |
+| `id`              | Server-generated request identifier                        |
+| `status`          | Inference status, typically `SUCCESS`                      |
+| `timestamp`       | Response timestamp                                         |
+| `processing_time` | End-to-end inference time in seconds                       |
+| `metadata`        | Deployment metadata for the model that handled the request |
+| `data`            | Capability-specific payload                                |
+
+## Python client
+
+```python
+import asyncio
+
+from pixano_inference.client import PixanoInferenceClient
+from pixano_inference.schemas import SegmentationRequest
+
+
+async def main() -> None:
+    client = PixanoInferenceClient.connect("http://localhost:7463")
+    request = SegmentationRequest(
+        model="sam2-image",
+        image="data:image/png;base64,...",
+        points=[[[200, 175]]],
+        labels=[[1]],
+    )
+    response = await client.segmentation(request)
+    print(response.processing_time)
+    print(response.data.scores.to_numpy())
+
+
+asyncio.run(main())
+```
+
+## Error responses
+
+- `400` when the model exists but does not support the requested capability.
+- `404` when the requested model name is not deployed.
+- `422` when the request body fails schema validation.
+- `500` when inference fails inside the model deployment.
+
+The Python client raises `fastapi.HTTPException` with the server error detail
+when a request is unsuccessful.

docs/README.md

@@ -8,58 +8,23 @@
 
 # Pixano Inference API reference
 
-## Client module
-
-The client module contains the class for the API client. It is responsible for making requests to the API endpoints.
-
-## Data module
-
-The data module contains the functions to read (and later write) data from/to a database or file.
-
-## Model registry module
-
-The model registry module contains the functions to register a model to the application.
-
-## Models module
-
-The models module contains the inference models to perform the tasks Pixano Inference API is designed for.
-
-The models include:
-
-- `BaseInferenceModel`: Base class for all Pixano Inference API models.
-- `Sam2Model`: Model used to detect and segment objects in images and videos.
-- `TransformerModel`: Model instantiated from Transformers.
-- `VLLMModel`: Model instantiated from VLLM.
-
-## Providers module
-
-The providers module contains the functions to load a model and to perform inference either from a model provider or from an API provider.
-
-The providers include:
-
-- `BaseProvider`: Base class for all Pixano Inference API providers.
-- `Sam2Provider`: Provider used to instantiate a `Sam2Model` and call its methods.
-- `TransformersProvider`: Provider used to instantiate a `TransformerModel` and call its methods.
-- `VLLMProvider`: Provider used to instantiate a `VLLMModel` and call its methods.
-
-## Pydantic module
-
-The pydantic module contains the classes for data validation. It is used by the models, providers, and the application itself to validate the input/output of the API.
-
-## Routers module
-
-The routers module contains the routers for the API. Each router has a path prefix that defines the endpoint where it will be mounted in the API.
-
-The routers swagger is accessible at at the `/docs` endpoint.
-
-## Settings module
-
-The settings module contains the configuration of the application.
-
-## Tasks module
-
-The tasks module contains the enums used to define the task that a model can perform.
-
-## Utils module
-
-The utils module contains the functions and classes used by the other modules.
+This section documents the public Python modules for the Ray Serve-based
+Pixano-Inference API.
+
+## Public modules
+
+- `pixano_inference.client`
+  Python client for the HTTP API.
+- `pixano_inference.configs`
+  Typed deployment configuration objects used in Python config files.
+- `pixano_inference.models`
+  Base classes, I/O models, and `register_model` for custom deployments.
+- `pixano_inference.ray`
+  Server bootstrap and Ray Serve integration.
+- `pixano_inference.schemas`
+  HTTP-layer request/response schemas and shared helper types.
+- `pixano_inference.settings`
+  Runtime settings exposed by the server.
+  Task enums and task-string helpers.
+- `pixano_inference.utils`
+  Shared helper utilities.