|
| 1 | +# AGENTS.md — Scylla Manager |
| 2 | + |
| 3 | +Scylla Manager (SM) is a management tool for Scylla clusters. |
| 4 | +Scylla cluster is a distributed database serving traffic via either CQL or Alternator frontend. |
| 5 | +The most important role of SM is to orchestrate repair, backup, restore tasks on managed Scylla cluster. |
| 6 | +SM consists of three binaries: `scylla-manager` (SM server), `scylla-manager-agent` (per-node SM agent), and `sctool` (CLI for interacting with SM server). |
| 7 | + |
| 8 | +## Project structure |
| 9 | + |
| 10 | +It consists of 5 go modules: |
| 11 | +- the main module (.) containing SM server and SM agent |
| 12 | +- backupspec module (./backupspec) containing backup directory specification |
| 13 | +- managerclient module (./v3/pkg/managerclient) containing client for SM server |
| 14 | +- swagger module (./v3/swagger) containing definitions of used Scylla, SM server, SM agent swagger endpoints |
| 15 | +- util module (./v3/pkg/util) containing generic legacy helpers (new helpers are added to ./pkg/util2) |
| 16 | + |
| 17 | +In case changes need to be made across multiple modules, the need to be merged in separate per module PRs. |
| 18 | +Modules in go.mod can only reference commits from master branch. |
| 19 | +All dependencies for main module (also other go modules from this repo) are vendored in /vendor dir. |
| 20 | +Documentation is stored in ./docs/source dir. |
| 21 | + |
| 22 | +### SM server architecture |
| 23 | + |
| 24 | +SM server is built around its scheduler responsible for scheduling and triggering tasks and services responsible |
| 25 | +for executing those tasks. |
| 26 | +Scheduler is implemented on two levels: |
| 27 | +- ./pkg/scheduler - implementation responsible for scheduling tasks at the right time |
| 28 | +- ./pkg/service/scheduler - implementation responsible for triggering tasks with all required information |
| 29 | + |
| 30 | +Services are implemented in ./pkg/service and the most important ones are: |
| 31 | +- ./pkg/service/backup - responsible for backup task |
| 32 | +- ./pkg/service/restore - responsible for restore task |
| 33 | +- ./pkg/service/repair - responsible for repair task |
| 34 | + |
| 35 | +The `sctool` CLI is implemented in ./pkg/command. |
| 36 | + |
| 37 | +A single SM server can manage multiple Scylla clusters. |
| 38 | +SM server stores cluster info, task definition and progress in a separate Scylla cluster (SM DB). |
| 39 | +It's usually local, single node cluster. SM server communicates with SM DB over CQL. |
| 40 | +SM server uses those information to start, pause, resume tasks and display their progress. |
| 41 | + |
| 42 | +User interacts with SM server via HTTP/S REST API defined in swagger module. |
| 43 | +To do so, user can use the `sctool` CLI or `managerclient` SDK. |
| 44 | + |
| 45 | +SM server communicates with managed Scylla clusters mainly via SM agent proxy. This communication is always required. |
| 46 | +In many other, but not all cases, SM server communicates with Scylla cluster directly over CQL. |
| 47 | +It requires adding cluster with specified CQL credentials. The same goes for Scylla clusters using Alternator frontend. |
| 48 | + |
| 49 | +### SM agent architecture |
| 50 | + |
| 51 | +SM agent is a small server running on every Scylla node. It serves as a proxy for SM server to Scylla HTTP REST API, |
| 52 | +which is exposed only on localhost. Additionally, SM agent also encapsulates rclone server. It is used for managing |
| 53 | +backup files on Scylla nodes and in backup locations (e.g., S3, GS, Azure, Minio, etc.). |
| 54 | +Communication with SM agent requires special auth token to be present in each request header. |
| 55 | + |
| 56 | +SM agent is implemented in ./pkg/cmd/agent and its rclone component is implemented in ./pkg/rclone. |
| 57 | + |
| 58 | +## General code guidelines |
| 59 | + |
| 60 | +When executing tasks, SM server does not perform any CPU, disk, network intensive operations. |
| 61 | +It serves as an orchestrator while Scylla node is doing all the heavy work. In case of backup |
| 62 | +and restore, SM agent is also responsible for moving files from backup location, which is network |
| 63 | +intensive operation. |
| 64 | +The only resource that might be limited to SM server is its memory. |
| 65 | +The rule of thumb is that SM server can keep all single node or table scope context needed for |
| 66 | +running given task (e.g., entire SM backup manifest with all files listed in it). |
| 67 | +Extra caution should be taken when storing entire cluster scope context in memory. |
| 68 | +In such cases, it's preferable to change the implementation to a per node or table context, |
| 69 | +unless this would result in less optimized usage of intensive Scylla node or SM agent API calls. |
| 70 | + |
| 71 | +SM project is in maintenance state - meaning that no new SM features are added, |
| 72 | +but only the changes needed to support new Scylla features are implemented. |
| 73 | +SM is also supposed to support all currently supported Scylla versions. |
| 74 | + |
| 75 | +Because of that, when working on SM server codebase, it's important to remember that: |
| 76 | +- changes to existing codebase should be minimal and backward compatible |
| 77 | +- new code should live mostly in new files and new packages |
| 78 | +- new code should optimize the usage of intensive Scylla node and SM agent API calls |
| 79 | +- new code should prioritize simplicity and readability over optimizing the usage of lightweight Scylla node and SM agent API calls |
| 80 | +- new code shouldn't optimize SM codebase itself |
| 81 | +- new code should take into consideration SM server memory consumption |
| 82 | + |
| 83 | +If new code introduces user facing changes, it should also adjust the documentation. |
| 84 | + |
| 85 | +## Build Commands |
| 86 | + |
| 87 | +```bash |
| 88 | +make build # Build all three binaries |
| 89 | +``` |
| 90 | + |
| 91 | +## Test Commands |
| 92 | + |
| 93 | +```bash |
| 94 | +# Unit tests (all packages or specific test) |
| 95 | +make unit-test |
| 96 | +make unit-test RUN=TestName |
| 97 | + |
| 98 | +# Integration tests (per pkg or specific test, require already set up test dev env) |
| 99 | +make pkg-integration-test PKG=./pkg/service/backup |
| 100 | +make pkg-integration-test PKG=./pkg/service/repair RUN=TestName |
| 101 | +``` |
| 102 | + |
| 103 | +## Lint and Format |
| 104 | + |
| 105 | +```bash |
| 106 | +make check # Full static analysis suite (should pass for every commit) |
| 107 | +``` |
| 108 | + |
| 109 | +## Development Environment |
| 110 | + |
| 111 | +```bash |
| 112 | +make start-dev-env SCYLLA_VERSION=<version> TABLETS=<enabled|disabled> SSL_ENABLED=<true|false> # Start test dev env (6-node Scylla cluster, another 2-node cluster MinIO, etc.) |
| 113 | +make run-server SSL_ENABLED=<true|false> # Build and run SM server in test dev env |
| 114 | +``` |
| 115 | + |
| 116 | +## Testing |
| 117 | + |
| 118 | +Code should be written with both unit tests and integration tests in mind. |
| 119 | +Integration tests should follow table-driven approach whenever possible, |
| 120 | +have separate setup and test execution stages and focus on comprehensiveness. |
| 121 | +For the changes to be approved, the following checks must pass: |
| 122 | +- make check |
| 123 | +- make unit-test |
| 124 | +- make pkg-integration-test PKG=<service path> RUN=<new test name> |
| 125 | + |
| 126 | +## Commit Messages |
| 127 | + |
| 128 | +Changes should be split into commits following Conventional Commits specification. |
0 commit comments