To restore from backup, Dremio services cannot be actively using the KV store directory for the platform. This article describes the process to activate a running pod with a claim on the volume storing the KV store, but without holding a lock on the KV store or running active coordinator services on it.
Switch cluster into Admin mode
The cluster needs to be switched into Admin-mode. If the coordinator node is running, there is a lock on the KV store and a restore will not work. The following command will stop the coordinator and launch an admin pod.
helm upgrade <cluster name> . -f ./values.yaml --set DremioAdmin=true [-n <my_namespace>]
Connect to the dremio-admin pod
Verify that the dremio-admin pod is available:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
dremio-admin 1/1 Running 0 98s
dremio-backup-adls-1626688800-jsrfr 0/1 Completed 0 36m
dremio-query-analyzer-1626675000-27lmp 0/1 Completed 0 55m
mongodb-1623769373-5cf98cd876-b4jgv 1/1 Running 0 53m
Run the following command to connect to the admin pod:
kubectl exec -i -t [-n <my_namespace>] dremio-admin -c dremio-admin "--" bash
Restore as usual:
Once logged into the admin pod you will need to rename the data/db directory and create a new empty data/db location. You can run the dremio-admin restore -d <path> as you normally would, specifying the location the backup is at. For specific guidance in various datalakes or other storage systems please open a Support Case.
Revert the admin pod change:
When the restore is complete, you can use Helm again to replace your master-coordinator pod, and take down the admin pod.
Run the following command to return your master-coordinator pod to normal:
helm upgrade <cluster name> . -f ./values.yaml --set DremioAdmin=false [-n <my_namespace>]