Document recommended Kubernetes maintenance workflows

[hammerhead]

Customers operating CrateDB through crate-operator need guidance for safely upgrading or replacing Kubernetes worker nodes.
In a recent support case, a customer reported unavailability of data (missing primary shards) during an upgrade of Kubernetes nodes. Even though such events aren't triggered by the operator, customers still seek advice from us as it concerns deployments created by the operator.

The operator currently defines a disruption budget with max_unavailable=1, meaning only one pod can be unavailable at a time:

crate-operator/crate/operator/create.py

Lines 1122 to 1160 in 2d05138

	def get_pod_disruption_budget(
	owner_references: Optional[List[V1OwnerReference]],
	name: str,
	node_name: str,
	node_name_prefix: str,
	) -> V1PodDisruptionBudget:
	"""
	Build the ``PodDisruptionBudget`` protecting one node spec's pods.
	A PodDisruptionBudget ensures that when performing Kubernetes cluster
	maintenance (i.e. upgrades), we make sure to not disrupt more than 1 pod in a
	StatefulSet at a time.
	The budget is named after the StatefulSet it guards, because
	``create_statefulset`` runs once per node spec and a cluster-wide name would
	collide: the first budget wins and every later one is dropped as a conflict,
	leaving those node specs unprotected (crate/cloud#3037).
	:param owner_references: Owner references to set on the budget.
	:param name: The CrateDB custom resource name defining the CrateDB cluster.
	:param node_name: The name of the node spec, e.g. ``master`` or ``hot``.
	:param node_name_prefix: The pod name prefix of the node spec, e.g.
	``master-`` or ``data-hot-``.
	"""
	return V1PodDisruptionBudget(
	metadata=V1ObjectMeta(
	name=f"crate-{node_name_prefix}{name}",
	owner_references=owner_references,
	),
	spec=V1PodDisruptionBudgetSpec(
	max_unavailable=1,
	selector=V1LabelSelector(
	match_labels={
	LABEL_COMPONENT: "cratedb",
	LABEL_NAME: name,
	LABEL_NODE_NAME: node_name,
	}
	),
	),

This ensures Kubernetes waits until an unavailable pod has become ready again, but a ready pod doesn't equal a ready CrateDB (all [primary] shards recovered). When strictly following this metric, data availability in CrateDB is not guaranteed.

For such externally initiated Kubernetes maintenance, there is no documentation explaining how to run in in a safe way, so that:

1. Kubernetes hosts are upgraded/replaced without CrateDB entering a red state.
2. No unnecessary shard recovery/rebalancing happens while Kubernetes nodes are restarted.
3. Users know how to determine when it is safe to drain the next Kubernetes host.

A documented procedure similar to the Rolling Upgrade Guide, addressing how to perform a Kubernetes upgrade safely, addressing above points.

---

How Elasticsearch approaches it

Elasticsearch dynamically updates the PDB:

In Elastic Cloud on Kubernetes clusters that do not have an Enterprise license, one Elasticsearch Pod can be taken down at a time, as long as the cluster has a health status of green. Single-node clusters are not considered highly available and can always be disrupted.

If a node is taken down, the PDB is updated to disallow the removal of additional pods. It changes the PDB back again once the cluster is in a green state.