feat(docs): backup, document --retention-lock-mode=event-based-hold

Fixes https://scylladb.atlassian.net/browse/CLOUD-3224

Author: Michal-Leszczynski

docs/source/backup/retention-lock.rst

@@ -3,7 +3,7 @@ Retention Lock
 ==============
 
 ScyllaDB Manager can protect snapshot files from accidental or malicious deletion
-by applying object-level retention locks on snapshot files stored in backup bucket.
+by applying object retention locks or event based holds on snapshot files stored in backup bucket.
 When retention lock is enabled, snapshot files in the backup location cannot be
 deleted until the retention period expires.
 
@@ -16,7 +16,10 @@ Retention lock is currently supported for :doc:`Google Cloud Storage <setup-gcs>
 How It Works
 ============
 
-When retention lock is enabled on a backup task, ScyllaDB Manager applies object-level retention
+Object retention lock modes (``unlocked``, ``locked``)
+------------------------------------------------------
+
+When object retention lock is enabled on a backup task, ScyllaDB Manager applies object-level retention
 to all snapshot files from given backup task execution. This includes schema files, SSTable files,
 and manifest files. The retention lock is applied during dedicated stage (``RETENTION_LOCK``) that
 runs after backup is finalized and all snapshot files are already in backup location.
@@ -27,14 +30,51 @@ contained in the snapshot tag, and lasts for the specified retention days.
 This means that a snapshot with ``--retention-days 30`` will have its files protected for exactly 30 days
 from when the snapshot was taken, regardless of how long the backup task takes to complete.
 
+This approach results in making a per-file request for both newly uploaded and deduplicated files.
+When making highly deduplicated backups to a colder storage tier (low storage costs, high request costs),
+consider using ``event-based-hold`` mode which avoids making additional requests for deduplicated files.
+
+Event based hold mode (``event-based-hold``)
+--------------------------------------------
+
+This mode utilizes the following cloud provider features for protecting snapshot files:
+
+* `Default bucket event based holds <https://docs.cloud.google.com/storage/docs/object-holds#default-holds>`_ - uploaded
+  objects have event based hold set. This hold needs to be removed before protected object can be deleted.
+* `Bucket retention lock <https://docs.cloud.google.com/storage/docs/bucket-lock>`_ - removing event based hold from an
+  object starts specified retention period during which the object can't be deleted.
+
+ScyllaDB Manager utilizes those bucket features in the following way:
+
+* Newly uploaded snapshot files automatically have event based hold applied.
+* Deduplicated snapshot files keep their hold.
+* Files referenced by previous snapshot which are not a part of the current one have their event based holds released. This starts their retention period.
+
+The retention period is configured via bucket configuration, not the ``--retention-days`` or ``--retention`` flags.
+ScyllaDB Manager won't attempt to purge stale snapshots according to backup task retention policy,
+if they are still protected by either event based holds or already started retention period.
+
+The main benefit of this approach is that it makes only a single per-file request and does not repeat those requests for deduplicated files.
+When making highly deduplicated backups to a colder storage tier (low storage costs, high request costs),
+it's possible that the request costs can dominate the overall backup costs.
+``event-based-hold`` mode aims to reduce costs in such scenarios.
+
+Note that since the holds for previous snapshot are released only during the backup task execution,
+corresponding files won't be removed from the backup storage for at least ``bucket_retention_period + backup_task_interval``.
+
+Note that because all objects uploaded to the bucket are subject to the default retention policy,
+files coming from aborted backups, temporary manifests and permission check files can't be removed
+until their holds are released and the retention period expires.
+
 Modes
 =====
 
-Retention lock supports three modes controlled by the :ref:`sctool backup --retention-lock-mode <sctool-backup>` flag:
+Retention lock supports the following modes controlled by the :ref:`sctool backup --retention-lock-mode <sctool-backup>` flag:
 
 * ``disabled`` (default): No retention lock is applied to snapshot files.
 * ``unlocked``: Retention lock is applied but can be shortened or removed with special permissions (see `Prerequisites`_).
 * ``locked``: Retention lock is applied and cannot be overridden. Once set, the lock cannot be removed or shortened, even by the bucket owner.
+* ``event-based-hold``: While referenced by the newest snapshot, files are protected by default event based holds. After that, they are protected by default retention period.
 
 Override Lock
 =============
@@ -56,19 +96,30 @@ Prerequisites
 
    .. group-tab:: Google Cloud Storage
 
-      .. rubric:: Bucket configuration
+      .. rubric:: Bucket configuration for ``unlocked`` and ``locked`` modes
 
       The GCS bucket used as the backup location must have **Object Retention** enabled.
       Refer to the `Enable and use object retention configurations documentation <https://docs.cloud.google.com/storage/docs/using-object-lock>`_
       for instructions on creating a bucket with Object Retention enabled.
 
+      .. rubric:: Bucket configuration for ``event-based-hold`` mode
+
+      The GCS bucket used as the backup location must have a **default retention policy** configured -
+      it defines the protection period of snapshot files (see `How It Works`_).
+      Refer to the `Use and lock retention policies <https://docs.cloud.google.com/storage/docs/using-bucket-lock#set-policy>`_
+      for instructions on setting a default retention policy on a bucket.
+
+      It is also recommended to enable the **default event based hold** option on the bucket,
+      so that the initial request setting the hold can be avoided.
+      Refer to the `Use object holds <https://docs.cloud.google.com/storage/docs/holding-objects#set-default-hold>`_ for details.
+
       .. rubric:: Permissions
 
       The GCS service account used by ScyllaDB Manager Agent must have the following permissions
       on the backup bucket:
 
       * ``storage.objects.update`` — required for updating object metadata.
-      * ``storage.objects.setRetention`` — required for applying retention locks to snapshot files.
+      * ``storage.objects.setRetention`` — required for applying retention locks to snapshot files in ``unlocked`` and ``locked`` modes.
       * ``storage.objects.overrideUnlockedRetention`` — required when using the ``--override-retention-lock`` flag
         to modify or remove existing locks in ``unlocked`` mode.
 
@@ -97,11 +148,18 @@ You can also :ref:`update an existing backup task <backup-update>` to enable ret
 
    sctool backup update -c <cluster ID> <backup task ID> --retention-lock-mode unlocked --retention-days 14
 
-Note that when retention lock is enabled:
+Note that when ``unlocked`` or ``locked`` retention lock mode is enabled:
 
 * ``--retention-days`` should be set to a positive value.
 * Count-based ``--retention`` should not be set.
 
+The ``event-based-hold`` mode can be combined with any retention policy, but the most intuitive
+configuration is to set ``--retention-days`` to the same value as the bucket's retention period:
+
+.. code-block:: none
+
+   sctool backup -c <cluster ID> -L gcs:<bucket> --retention-lock-mode event-based-hold --retention-days <bucket retention period days>
+
 Changing retention lock configuration
 -------------------------------------
 
@@ -115,6 +173,14 @@ deviating from them may lead to errors during the purge stage (see `Shared files
 * In ``locked`` mode, ``--retention-days`` should only be **increased**, not decreased.
 * Decreasing ``--retention-days`` in ``unlocked`` mode should be accompanied by the ``--override-retention-lock`` flag (see `Override Lock`_).
 
+.. warning::
+
+   Changing the retention lock mode between ``unlocked``/``locked`` and ``event-based-hold``
+   (in either direction) is **not supported**. These approaches to protecting snapshot files
+   are not compatible with each other and require different bucket configurations.
+   To switch between them, create a new backup task pointing to a different backup location
+   configured for the desired mode.
+
 Shared files
 ============