Summary
This article describes the process to activate a running pod with a claim on the volume storing the KV store, but without holding a lock on the KV store or running active coordinator services on it, in order to restore from backup.
Reported Issue
To restore from backup, Dremio services cannot be actively using the KV store directory for the platform.
Overview
The cluster needs to be switched into Admin-mode to stop the coordinator and launch an admin pod, as the coordinator node running would have a lock on the KV store, preventing a restore from working.
Relevant Versions Tools and Integrations
All currently supported versions of Dremio Enterprise Edition launched with Dremio V2 Helm charts, found at: https://github.com/dremio/dremio-cloud-tools/tree/master/charts/dremio_v2
Steps to Resolve
- Switch cluster into Admin mode using the `helm upgrade` with `--set DremioAdmin=true` command to stop the coordinator and launch an admin pod.
- Connect to the dremio-admin pod using the `kubectl exec` command.
- Restore as usual by renaming the current data/db directory, creating a new empty data/db location, and running `dremio-admin restore -d` pointing to the new data/db location
- Revert the admin pod change using `helm upgrade` with `--set DremioAdmin=false` to replace the master-coordinator pod and take down the admin pod.