Would like to provide a best pratics for AWS PV Migration

[Eroyi]

Not sure if it's worth creating a pull request to put on the USAGE.md, i might just leave it here if you find this is helpful.

For migrating PV on AWS, there are a few additional configurations required:

1. Amazon EKS requires a specific annotation on service.yaml to create a valid LoadBalancer ( nlb / alb / slb ). Check which kind of Load Balancer you are using.
2. After LB is created, the FQDN on the External IP would take some time to be answerable.
3. Service behind the FQDN needs to be connectable, and in most cases, it is. But if you applied additional security or network isolation config, you may want to lift those restrictions temporarily.

Command:

pv-migrate migrate \
--source-kubeconfig /root/.kube/config \
--source-context arn:aws:eks:ap-southeast-1:000000000000cluster/non-prod \
--source-namespace mongodb \
  # 👆 Specify the source cluster

--dest-kubeconfig /root/.kube/config \
--dest-context arn:aws:eks:ap-southeast-1:000000000000:cluster/dev \
--dest-namespace mongodb \
  # 👆 Specify the destination cluster

--helm-set \
sshd\.service\.annotations\."service\.beta\.kubernetes\.io\/aws-load-balancer-nlb-target-type"="instance",\
sshd\.service\.annotations\."service\.beta\.kubernetes\.io\/aws-load-balancer-name"="pv-migrate",\
sshd\.service\.annotations\."service\.beta\.kubernetes\.io\/aws-load-balancer-scheme"="internet-facing",\
sshd\.service\.annotations\."service\.beta\.kubernetes\.io\/aws-load-balancer-type"="external",\
  # 👆 To create a service with the correct ExternalIP ( FQDN ), append these configurations 
        on sshd.service.annotation gives it the ability to talk with the aws-load balancer plugin.

rsync.maxRetries=20,\
rsync.retryPeriodSeconds=60 \
  # 👆 FQDN registration could take a while before it can be resolved correctly, usually within 5-10 min. 
        Default retries are 10 times with 5 seconds period, which is too fast, 
        makes the whole migration process fail before FQDN becomes valid.

--ignore-mounted \
  # 👆 Set this flag if your database or service is fully disconnected from any activity, 
        yet you are not allowed to scale down/delete those pods.

--log-level debug --log-format json \
--dest-delete-extraneous-files \
  # 👆 This flag will eradicate all files on the DEST directory; only set it when you know what you are doing

--strategies lbsvc \
  # 👆 There is no point in using any other strategies since we are doing this with the LoadBalancer strategy

--helm-timeout 10m \
  # 👆 Sometimes EKS API could take a little longer to create a resource while under heavy traffic loads,
        the default is 1m

non-prod-mongodb-pv dev-mongodb-pv
  # 👆 Specify the old (source) pvc and the new (destination) PVC

The correct ExternalIP should look like this 👇

[luis-alen]

I just want to add that the default ELB idle connection timeout is 60s and that might cause rsync to lose its connection, especially when re-syncing a huge directory tree with thousands of files. In that case it's also worth adding an annotation to change it to a value at least greater than the sshd pod ClientAliveInternal setting of 300s. I'd recommend setting it to 1h to be on the safe side. The annotation that helped me achieve this was service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"

Please see: #320 (comment)

[utkuozdemir]

Thank you both, this information should definitely land in a documentation page, not sure which one yet (probably a separate one than USAGE.md though, with a link to it from USAGE.md)