feat(docs): improve backup and restore documentation (#115)

* feat(docs): improve backup and restore documentation

Author: bsctl

README.md

@@ -11,10 +11,10 @@ A multi-tenant deployment for `etcd` is not common practice. However, `etcd` pro
 ## Documentation
 Refer to the [etcd documentation](https://etcd.io/docs/v3.5/op-guide). Following sections provide additional procedures to help with a specific setup as it is used into project [Kamaji](https://github.com/clastix/kamaji).
 
+- [Inspecting the Cluster](docs/inspect.md)
 - [Monitoring](docs/monitoring.md)
-- [Taking Snapshots](docs/snapshot.md)
-- [Recover from Snapshot](docs/snapshot-recovery.md)
-- [Velero](docs/velero.md)
+- [Backup](docs/backup.md)
+- [Restore](docs/restore.md)
 - [Rotate Certificates](docs/rotate-certificates.md)
 - [Defragmenting Data](docs/defragmentation.md)
 - [Performance and Optimization](docs/performance-and-optimization.md)

docs/backup.md

@@ -0,0 +1,96 @@
+# Take a backup
+
+The `backup.sh` script is designed to create a job for taking snapshot of `etcd` instance. The script generates Kubernetes Job manifests and applies them to the specified namespace.
+
+## Overview
+The script performs the following steps:
+
+1. Creates a Kubernetes Job manifests from one of the `etcd` members.
+2. The job takes a snapshot of the `etcd` member and uploads it to a s3-like storage.
+
+## Prerequisites
+
+- Ensure you have `kubectl` installed and configured to interact with the management cluster.
+- It is assumed that the snapshot files will be stored on an S3-like storage.
+- A kubernetes secret called `backup-storage-secret` containing the parameters and credentials to access the storage must be created in the same namespace where `kamaji-etcd` is running.
+
+### Creating the Secret
+
+To create the secret, use the following command:
+
+```bash
+kubectl create secret generic backup-storage-secret \
+  --from-literal=storage-url=<storage_url> \
+  --from-literal=storage-access-key=<access_key> \
+  --from-literal=storage-secret-key=<access_secret> \
+  --from-literal=storage-bucket-name=<bucket_name> \
+  --from-literal=storage-bucket-folder=<bucket_folder> \
+  -n <etcd_namespace>
+```
+
+## Usage
+To run the script, use the following command:
+
+```bash
+./backup.sh [-e etcd_name] [-s etcd_client_service] [-n etcd_namespace]
+```
+
+### Parameters
+
+- `-e etcd_name`: Name of the etcd StatefulSet (default: `kamaji-etcd`)
+- `-s etcd_client_service`: Name of the etcd client service (default: `kamaji-etcd-client`)
+- `-n etcd_namespace`: Namespace of the etcd StatefulSet (default: `kamaji-system`)
+
+### Example
+
+To run the script with custom parameters:
+
+```bash
+./backup.sh -e kamaji-etcd -s kamaji-etcd-client -n kamaji-system
+```
+
+This will create a Kubernetes Job manifest with the specified parameters and apply it to the cluster. The job will take a snapshot of the `etcd` member and upload it to the specified S3-like storage.
+
+### Notes
+
+- Ensure you have access a s3-like storage and the necessary secret `backup-storage-secret` is configured in Kubernetes.
+- The script uses `kubectl` commands, so ensure you have the necessary permissions to perform these operations.
+
+### Debug mode
+To run the script in debug mode set the environment variable `DEBUG`:
+
+``` bash
+export DEBUG=1
+```
+
+## Schedule recurring backups
+To schedule recurring backups with CronJobs, use the `schedule.sh` script. The script generates Kubernetes CronJob manifests and applies them to the specified namespace.
+
+### Usage
+To run the script, use the following command:
+
+``` bash
+./schedule.sh [-e etcd_name] [-s etcd_client_service] [-n etcd_namespace] [-j schedule]
+``` 
+
+### Parameters
+
+- `-e etcd_name`: Name of the etcd StatefulSet (default: `kamaji-etcd`)
+- `-s etcd_client_service`: Name of the etcd client service (default: `kamaji-etcd-client`)
+- `-n etcd_namespace`: Namespace of the etcd StatefulSet (default: `kamaji-system`)
+- `-j schedule`: Cron schedule for the backup job (default: `"0 0 * * *"`, which means daily at midnight)
+
+### Example
+To run the script with custom parameters:
+
+```bash
+./schedule.sh -e kamaji-etcd -s kamaji-etcd-client -n kamaji-system -j "14 9 * * 1-5"
+```
+This will create a Kubernetes CronJob manifest with the specified parameters and apply it to the cluster. The CronJob will take a snapshot of the `etcd` member and upload it to the specified S3-like storage according to the defined schedule.
+
+### Debug mode
+To run the script in debug mode set the environment variable `DEBUG`:
+
+``` bash
+export DEBUG=1
+```

docs/inspect.md

@@ -0,0 +1,60 @@
+# Inspecting etcd Cluster
+This guide explains how to run a `etcd` client pod to inspect the `kamaji-etcd` cluster and perform various operations.
+
+## Running the etcd Client Pod
+
+The `etcd-client.sh` script in the `scripts` directory can be used to create an etcd client pod with all necessary configurations and certificates mounted. Here's how to use it:
+
+```bash
+./etcd-client.sh [-e etcd_name] [-s etcd_client_service] [-n etcd_namespace]
+```
+
+### Script Parameters
+
+- `-e`: Name of the etcd cluster (default: "kamaji-etcd")
+- `-s`: Service name for etcd client connection (default: "kamaji-etcd-client")
+- `-n`: Namespace where etcd is running (default: "kamaji-system")
+
+### Using the Client Pod
+
+1. Run the script to create the client pod:
+   ```bash
+   ./etcd-client.sh -e kamaji-etcd -s kamaji-etcd-client -n kamaji-system
+   ```
+
+2. Once the pod is created, connect to it using:
+   ```bash
+   kubectl exec -it etcd-client -n kamaji-system -- bash
+   ```
+
+3. Inside the pod, you can run various etcdctl commands:
+   - List all etcd cluster members:
+     ```bash
+     etcdctl member list -w table
+     ```
+   - Check endpoint status:
+     ```bash
+     etcdctl endpoint status -w table
+     ```
+   - Check endpoint health:
+     ```bash
+     etcdctl endpoint health -w table
+     ```
+   - List all keys in etcd:
+     ```bash
+     etcdctl get / --prefix --keys-only
+     ```
+
+The client pod is pre-configured with:
+
+- TLS certificates for secure communication
+- Correct endpoints configuration
+- Required environment variables
+
+### Cleanup
+
+To remove the client pod when finished:
+```bash
+kubectl delete pod etcd-client -n kamaji-system
+```
+This allows you to inspect and interact with the `kamaji-etcd` cluster without needing to install etcdctl on your local machine or manage certificates manually.

docs/snapshot-recovery.md docs/restore.mddocs/snapshot-recovery.md renamed to docs/restore.md

@@ -40,25 +40,27 @@ To run the script, use the following command:
 ./restore.sh [-e etcd_name] [-s etcd_service] [-n etcd_namespace] [-f snapshot]
 ```
 
-## Parameters
+### Parameters
 
 - `-e etcd_name`: Name of the etcd StatefulSet (default: `kamaji-etcd`)
-- `-s etcd_service`: Name of the etcd service (default: `kamaji-etcd`)
+- `-s etcd_service`: Name of the etcd headless service (default: `kamaji-etcd`)
 - `-n etcd_namespace`: Namespace of the etcd StatefulSet (default: `kamaji-system`)
 - `-f snapshot`: Snapshot file to restore from (required)
 
-For example:
+### Example:
 
 ```bash
-./restore.sh -e my-etcd -s my-etcd-service -n my-namespace -f snapshot.db
+./restore.sh -e kamaji-etcd -s kamaji-etcd -n kamaji-system -f snapshot.db
 ```
 
-## Notes
+> 🚨 Make sure to use the **headless service** name for the `-s` parameter, which is typically the same as the StatefulSet name.
 
-- Ensure that the snapshot file is accessible and the necessary secret for accessing the storage is configured in the same namespace.
+### Notes
+
+- Ensure that the snapshot file is accessible and the necessary secret `backup-storage-secret` for accessing the storage is configured in the same namespace.
 - The script uses `kubectl` commands, so ensure you have the necessary permissions to perform these operations.
 
-## Debug mode
+### Debug mode
 To run the script in debug mode set the environment variable `DEBUG`:
 
 ``` bash