Addon: run Rook-Ceph inside Minikube

[bhavyaBeliever]

Steps to run:

1. Start a minikube cluster with disk-size atleast 40gb, 4 cpus and 8gb of memory:

./out/minikube start --disk-size=40g  --memory=8000 --cpus=4

2. Configure the rook-ceph addon with: (Optional)

./out/minikube addons configure rook-ceph

Two things that can be configured.

• The default (loop) disk size which should be atleast 6 GiB
• Device that can be used by Ceph (e.g. /dev/vdb), this needs to be created at the start time with flag extra-disks

3. Enable the addon, it takes few minutes. (majority of the time in pulling images)

./out/minikube addons enable rook-ceph

4. Verify the addon with:

minikube kubectl -- get cephcluster -n rook-ceph

Check if OSD pod is running with:

minikube kubectl -- get pods -n rook-ceph

Current Implementation Details and configuration of Ceph cluster:

1. There is only one disk that ceph cluster uses. The default disk is a loop device inside the VM.
2. No replication, all components like mon, mgr, osd are of count 1.
3. Two storage interfaces: RBD(works for VM-based drivers) and CephFS(works for all drivers).
4. Ceph Dashboard is enabled.
5. Two storage classes: rook-cephfs, rook-ceph-block.
6. Ceph runs only on one node since it runs only on one disk.
7. The dataDirHostPath is kept /data/rook instead of default /var/lib/rook, supports persistant storage.
8. minikube addons disable rook-ceph is implemented, mapped custom teardown callback, used steps that were mentioned here.

Pending Work:

• Multiple Disks / OSDs Support
• Object Storage (S3 API) Support
• Image Caching Strategy
• Support RBD storage in KIC-based driver
• Adding Testcases
• Enable CSIAddons & VolumeReplication
• Use New version of Rook-ceph: Currently using v16.7

fixes #21527

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: bhavyaBeliever
Once this PR has been reviewed and has the lgtm label, please assign medyagh for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Hi @bhavyaBeliever. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[minikube-bot]

Can one of the admins verify this patch?

[bhavyaBeliever]

@medyagh @prezha @nirs @obnoxxx would like to get feedback from your side!!

[nirs]

@bhavyaBeliever nobody can review 17k lines change, and the discussion on the issue barely started. We need to finish planning first before spending time on implementation.

[bhavyaBeliever]

@nirs Out of 17k lines much of this is yaml files that are copied from the rook repository(crds.yaml, common.yaml, operator.yaml). The rook-ceph-crds.yaml and rook-ceph-operator.yaml are almost 16k

[bhavyaBeliever]

Also, as we have taken this yaml files from rook repo what should be the license for such files?

[nirs]

@nirs Out of 17k lines much of this is yaml files that are copied from the rook repository

1k is much better than 17k but still a lot when we do't have any plan to discuss.

I would not add 16k of rook resources to minikube. It does not make sense to keep this in minikube. We depend on rook images from the registry so we can depend on pulling rook resources from GitHub in the same way.

Think abut updating rook version - the addon must update when rook updates. Pulling the resources from GitHub using kustomization makes update easier - you change the URL and maybe update the kustomization for new options added in the release.

I think the modern way to implement addons is helm, which should be supported now in rook. This will keep the minikube code minimal.

But the big question is not how to implement the addon, but why. Before we discuss the use cases we want to support there is no point to work on the implementation.

[bhavyaBeliever]

But the big question is not how to implement the addon, but why. Before we discuss the use cases we want to support there is no point to work on the implementation.

Agreed, The whole point of making this PR is to test different configuration we might be looking forward with use cases we wanted inside minikube cluster by maybe just modify the rook-ceph-cluster.yaml.