Merge pull request ait-detectmate#14 from annaerdi/docs

Add documentation

Author: annaerdi

.github/workflows/publish-docs.yml

@@ -0,0 +1,38 @@
+name: Publish Docs
+on:
+  release:
+    types: [published]
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Set up uv (Python 3.12)
+        uses: astral-sh/setup-uv@v5
+        with:
+          python-version: "3.12"
+          enable-cache: true
+
+      - name: Install dependencies
+        run: uv pip install -e .[dev]
+
+      - name: Configure Git user
+        run: |
+          git config --global user.name "${{ github.actor }}"
+          git config --global user.email "${{ github.actor }}@users.noreply.github.com"
+
+      - name: Deploy docs
+        run: |
+          # Extract version from tag (remove 'v' prefix if present)
+          VERSION=${GITHUB_REF_NAME#v}
+          echo "Deploying version $VERSION"
+          uv run mike deploy --push --update-aliases $VERSION latest
+          # Optional: make the released version the default shown when visiting the site
+          # it's enough to run this once
+          # uv run mike set-default --push latest

README.md

@@ -58,7 +58,7 @@ service = DemoService()
 
 with service:
     with pynng.Req0(dial=service.settings.manager_addr) as req:
-        for cmd in ("ping", "status", "pause", "status", "resume", "status", "stop"):
+        for cmd in ("ping", "status", "stop"):
             print(f">>> {cmd}")
             req.send(cmd.encode("utf-8"))
             reply = req.recv().decode("utf-8", "ignore")
@@ -75,17 +75,7 @@ Example configuration files can be found in the `tests/config` directory.
 Start the service:
 
 ```bash
-detectmate start --settings tests/config/service_settings.yaml --config tests/config/detector_config.yaml
-```
-
-Reconfigure the service:
-
-```bash
-# Update parameters in memory only
-detectmate reconfigure --settings tests/config/service_settings.yaml --config tests/config/new_config.yaml
-
-# Update parameters and persist to file. This overwrites the originally provided config file.
-detectmate reconfigure --settings tests/config/service_settings.yaml --config tests/config/new_config.yaml --persist
+detectmate start --settings tests/config/service_settings.yaml
 ```
 
 Get the service status:

demo/docker-compose.yml

@@ -3,8 +3,6 @@ services:
     build:
       context: ..
       dockerfile: demo/Dockerfile
-      args:
-        GITHUB_TOKEN: ${GITHUB_TOKEN}  # must be set in .env or export
     container_name: detectmate_reader
     command: >
       detectmate start
@@ -22,8 +20,6 @@ services:
     build:
       context: ..
       dockerfile: demo/Dockerfile
-      args:
-        GITHUB_TOKEN: ${GITHUB_TOKEN}
     container_name: detectmate_parser
     command: >
       detectmate start
@@ -44,8 +40,6 @@ services:
     build:
       context: ..
       dockerfile: demo/Dockerfile
-      args:
-        GITHUB_TOKEN: ${GITHUB_TOKEN}
     container_name: detectmate_detector
     command: >
       detectmate start
@@ -65,8 +59,6 @@ services:
     build:
       context: ..
       dockerfile: demo/Dockerfile
-      args:
-        GITHUB_TOKEN: ${GITHUB_TOKEN}
     container_name: detectmate_demo
     command: python demo/manual_demo_run_tcp.py
     volumes:

docs/configuration.md

@@ -0,0 +1,78 @@
+# Configuration
+
+DetectMateService can be configured using a YAML settings file or environment variables. Environment variables take precedence over the YAML file.
+
+## Service settings
+
+These settings control the service infrastructure.
+
+| Setting                       | Env Variable                             | Default                            | Description                                                                                               |
+| :---------------------------- | :--------------------------------------- | :--------------------------------- |:----------------------------------------------------------------------------------------------------------|
+| `component_name`              | `DETECTMATE_COMPONENT_NAME`              | `None`                             | A human-readable name for the service instance.                                                           |
+| `component_id`                | `DETECTMATE_COMPONENT_ID`                | `None` (computed)                  | Unique identifier for the component; computed automatically if not provided.                              |
+| `component_type`              | `DETECTMATE_COMPONENT_TYPE`              | `core`                             | Python import path for the component class (e.g., `detectors.MyDetector`).                                |
+| `component_config_class`      | `DETECTMATE_COMPONENT_CONFIG_CLASS`      | `None`                             | Python import path of the configuration class used by the component (e.g., `detectors.MyDetectorConfig`). |
+| `log_level`                   | `DETECTMATE_LOG_LEVEL`                   | `INFO`                             | Logging level (`DEBUG`, `INFO`, `WARNING`, `ERROR`).                                                      |
+| `log_dir`                     | `DETECTMATE_LOG_DIR`                     | `./logs`                           | Directory for log files.                                                                                  |
+| `log_to_console`              | `DETECTMATE_LOG_TO_CONSOLE`              | `true`                             | Whether logs are written to stdout/stderr.                                                                |
+| `log_to_file`                 | `DETECTMATE_LOG_TO_FILE`                 | `true`                             | Whether logs are written to files in `log_dir`.                                                           |
+| `manager_addr`                | `DETECTMATE_MANAGER_ADDR`                | `ipc:///tmp/detectmate.cmd.ipc`    | Address for management commands (REQ/REP).                                                                |
+| `manager_recv_timeout`        | `DETECTMATE_MANAGER_RECV_TIMEOUT`        | `100`                              | Receive timeout (ms) for the manager command channel.                                                     |
+| `manager_thread_join_timeout` | `DETECTMATE_MANAGER_THREAD_JOIN_TIMEOUT` | `1.0`                              | Timeout (s) when waiting for the manager thread to stop.                                                  |
+| `engine_addr`                 | `DETECTMATE_ENGINE_ADDR`                 | `ipc:///tmp/detectmate.engine.ipc` | Address for data processing (PAIR0/1).                                                                    |
+| `engine_autostart`            | `DETECTMATE_ENGINE_AUTOSTART`            | `true`                             | Whether the engine channel is started automatically.                                                      |
+| `engine_recv_timeout`         | `DETECTMATE_ENGINE_RECV_TIMEOUT`         | `100`                              | Receive timeout (ms) for the engine channel.                                                              |
+| `out_addr`                    | `DETECTMATE_OUT_ADDR`                    | `[]`                               | List of output addresses (strongly typed NNG URLs).                                                       |
+| `out_dial_timeout`            | `DETECTMATE_OUT_DIAL_TIMEOUT`            | `1000`                             | Timeout (ms) for connecting to output addresses.                                                          |
+
+
+### YAML files
+
+You can provide a YAML file containing the service settings. Below is an example `settings.yaml`:
+
+```yaml
+component_name: "my-detector"
+log_level: "DEBUG"
+log_dir: "./logs"
+
+# Manager Interface (Command Channel)
+manager_addr: "ipc:///tmp/detectmate.cmd.ipc"
+
+# Engine Interface (Data Channel)
+engine_addr: "ipc:///tmp/detectmate.engine.ipc"
+engine_autostart: true
+
+# Output Destinations (where processed data is sent)
+out_addr:
+  - "tcp://127.0.0.1:5000"
+  - "ipc:///tmp/output.ipc"
+
+out_dial_timeout: 1000
+```
+
+
+### Environment variables
+
+Environment variables override values in the YAML file. They are prefixed with `DETECTMATE_`.
+
+Example:
+```bash
+export DETECTMATE_LOG_LEVEL=DEBUG
+export DETECTMATE_COMPONENT_NAME=worker-1
+detectmate start
+```
+
+## Component configuration
+
+In addition to the service settings (which configure the *runner*), you can also pass a separate configuration file for the specific component logic (e.g., detector parameters) using the `--config` flag in the CLI. This file is specific to the implementation of the component you are running.
+
+
+
+```yaml
+# detector-config.yaml
+threshold: 0.85
+sensitivity: high
+enabled: true
+```
+
+You can read more about Components in the [Using a Library Component](library.md) section.

docs/index.md

@@ -0,0 +1,19 @@
+# DetectMate Service Framework
+
+Welcome to the DetectMate Service Framework documentation. DetectMate is a flexible, component-based framework for
+building distributed detection and processing services.
+
+It uses NNG's messaging architecture to process data in real-time.
+
+## Key features
+
+- **Modular design**: easily extensible with custom processors and components.
+- **Resilient networking**: built on top of [`pynng`](https://pynng.readthedocs.io/en/latest/) (NNG) for high-performance messaging.
+- **Configurable**: fully configurable via YAML files or environment variables.
+- **Service management**: built-in CLI for starting, stopping, and monitoring the service.
+- **Scalable**: run multiple independent service instances.
+
+## Getting started
+
+Check out the [Installation](installation.md) guide to set up the service, and then proceed to
+[Configuration](configuration.md) and [Usage](usage.md) to learn how to run it.

docs/installation.md

@@ -0,0 +1,55 @@
+# Installation
+
+First, clone DetectMateService and navigate into the repository:
+
+```bash
+git clone https://github.com/ait-detectmate/DetectMateService.git
+cd DetectMateService
+```
+
+## Setup with uv (recommended)
+
+We recommend using [uv](https://github.com/astral-sh/uv) to manage the environment
+and dependencies.
+
+### 1. Create and activate a virtual environment with uv
+
+```bash
+uv venv
+source .venv/bin/activate
+```
+
+### 2. Install the project
+
+```bash
+uv pip install .
+```
+
+## Alternative setup with pip
+
+If you prefer plain `pip`, you can set things up like this instead:
+
+```bash
+# Create a virtual environment
+python -m venv .venv
+# Activate it
+source .venv/bin/activate
+# Install the project in editable mode with dev dependencies
+pip install .
+```
+
+
+## Developer setup
+
+For development, you can install with optional dependencies:
+
+```bash
+pip install -e ".[dev]"
+```
+
+We recommend using [`prek`](https://github.com/j178/prek) to manage Git
+pre-commit hooks. `prek` is configured via the existing `.pre-commit-config.yaml`
+and can be installed as part of the `dev` extras. To ensure pre-commit hooks run before each commit, run:
+```bash
+prek install
+```

docs/library.md

@@ -0,0 +1,62 @@
+## Using a Library Component
+
+The Service can be run as any component imported from the [DetectMateLibrary](https://github.com/ait-detectmate/DetectMateLibrary).
+For this, ensure that the library is installed in the same activated virtual environment, where the service is installed.
+
+### 1. Update settings
+
+Modify `settings.yaml` to use a library component:
+
+```yaml
+component_name: random-detector
+component_type: detectors.RandomDetector
+component_config_class: detectors.RandomDetectorConfig
+config_file: detector-config.yaml
+log_level: INFO
+manager_addr: ipc:///tmp/detectmate.cmd.ipc
+engine_addr: ipc:///tmp/detectmate.engine.ipc
+```
+
+### 2. Create component configuration
+
+Create `detector-config.yaml`:
+
+```yaml
+threshold: 0.75
+window_size: 10
+enabled: true
+```
+
+### 3. Start with configuration
+
+```bash
+detectmate start --settings settings.yaml --config detector-config.yaml
+```
+
+### 4. Reconfigure at runtime
+
+Create `new-config.yaml`:
+
+```yaml
+threshold: 0.85
+window_size: 15
+enabled: true
+```
+
+The service supports dynamic reconfiguration with two modes:
+
+#### 1. In-memory update (default)
+Changes are applied to the running service but not saved to disk. The changes will be lost when the service restarts.
+
+```bash
+detectmate reconfigure --settings settings.yaml --config new-config.yaml
+```
+
+#### 2. Persistent update (with --persist flag)
+Changes are applied to the running service AND saved to the original parameter file. The changes persist across service restarts.
+
+```bash
+detectmate reconfigure --settings settings.yaml --config new-config.yaml --persist
+```
+
+**Note:** The `--persist` flag will overwrite the original parameter file specified in your service configuration with the new values from the `--params` file.

docs/pipeline.md

@@ -0,0 +1,63 @@
+# Running the Docker Pipeline
+
+There is a containerized demonstration of the DetectMate log analysis pipeline. The demo runs three services (reader, parser,
+detector) that process audit logs to detect anomalies, with a test script that feeds log lines through the complete
+pipeline and reports detected anomalies. Each component runs in its own Docker container and the components communicate via TCP.
+
+This document explains how to run the example pipeline using Docker, located in the `demo` folder.
+
+
+## Components
+
+The pipeline consists of the following services defined in `demo/docker-compose.yml`:
+
+1.  **Reader (`detectmate_reader`)**:
+    *   reads log lines from a source file (`demo/data/audit.log`)
+    *   uses `demo/config/reader_config.yaml`
+    *   listens on internal port **8001**
+
+2.  **Parser (`detectmate_parser`)**:
+    *   receives raw log data, parses it into a structured format
+    *   uses `demo/config/parser_config.yaml`
+    *   listens on internal port **8011**
+    *   depends on the Reader
+
+3.  **Detector (`detectmate_detector`)**:
+    *   analyzes structured logs to detect anomalies
+    *   uses `demo/config/detector_config.yaml`
+    *   listens on internal port **8021**
+    *   depends on the Parser
+
+4.  **Demo Driver (`detectmate_demo`)**:
+    *   Acts as the orchestrator/client. It executes the `demo/manual_demo_run_tcp.py` script. It does the following:
+        1. count lines in the audit log
+        2. for each line, it sends a request to the **Reader** to get a log line
+        3. forwards the log line to the **Parser** for processing
+        4. forwards the parsed data to the **Detector** for analysis
+        5. prints any detected anomalies to the console
+
+
+## How to Run
+
+Prerequisites: **Docker** and **Docker Compose** installed.
+
+**Terminal 1** (keep running to see service logs):
+```bash
+cd demo
+docker compose up reader parser detector
+```
+
+**Terminal 2** (run after services are up):
+```bash
+cd demo
+docker compose up demo
+```
+
+
+## Directory Structure
+
+*   `demo/Dockerfile`: The Docker definition used by all services.
+*   `demo/docker-compose.yml`: Defines the multi-container application.
+*   `demo/config/`: Contains YAML configuration files for Reader, Parser, and Detector.
+*   `demo/data/`: Contains sample data (e.g., `audit.log`) used by the Reader.
+*   `demo/manual_demo_run_tcp.py`: The Python script running inside the `demo` container that drives the flow.