Overview

In a Dremio deployment on Kubernetes, it requires extra handling to grow the /opt/dremio/data mount point which the dremio-master-0 depends upon. This article will detail the process of increasing space for the volume in the deployment.

Applies To

Any Kubernetes-based deployment of Dremio.

Details

A Dremio cluster depends on an internal database to hold the catalog and other internal data. This database will grow over time with usage of Dremio, and it may be required at some point to increase the size of the storage volume the database is mounted to.

In a Kubernetes deployment, the dremio-master-0 pod attaches a Persistent Volume Claim (PVC) mapped to a StatefulSet (STS,) allowing restarts of the Dremio master to reliably reattach the same storage volume between restarts. This typically mounts to /opt/dremio/data in the Dremio master pod.

Due to a limitation in Kubernetes StatefulSet code, it’s not possible to directly increase the size of the PVC with a change to the definition of the STS. You will see a message like this:

spec: Forbidden: updates to statefulset spec for fields other than ‘replicas’, ‘template’, and ‘updateStrategy’ are forbidden

To increase the size of the volume, you can perform the following steps.

Prequisite:

Ensure the PVC/PV “StorageClass” supports “AllowVolumeExpansion” and it is set to “True.”

1. Check the PVC details with kubectl get pvc [-n <namespace>] command. In the below example the StorageClass name is “default”:

devin [ ~/dremio-cloud-tools/charts/dremio_v2 ]$ kubectl get pvc
NAME                                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
datadir-zk-0                                       Bound    pvc-9ee5eaae-d0be-4407-8e1d-d17dfa21ea84   10Gi       RWO            default        103m
datadir-zk-1                                       Bound    pvc-3618538b-e66c-4573-9d3c-2d2a9a8f35e2   10Gi       RWO            default        103m
datadir-zk-2                                       Bound    pvc-84adf868-1a79-43df-a0f6-71dec5720ab1   10Gi       RWO            default        103m
dremio-default-executor-c3-0-dremio-executor-0     Bound    pvc-4334605d-d345-46c2-b66a-e426772b27a0   100Gi      RWO            default        103m
dremio-default-executor-volume-dremio-executor-0   Bound    pvc-4ad517a9-b8f4-4c4a-b599-96a2608868e4   128Gi      RWO            default        103m
dremio-master-volume-dremio-master-0               Bound    pvc-b843efb1-0f6f-42ac-8195-9d3cb0e81974   721Gi      RWO            default        103m

2. Then get the description of that “StorageClass” with kubectl describe StorageClass <storage class name> [-n <namespace>] which in this example will result in the command kubectl describe StorageClass default and in my cluster it returned the following response:

devin [ ~/dremio-cloud-tools/charts/dremio_v2 ]$ kubectl describe StorageClass default
Name:                  default
IsDefaultClass:        Yes
Annotations:           storageclass.kubernetes.io/is-default-class=true
Provisioner:           disk.csi.azure.com
Parameters:            skuname=StandardSSD_LRS
AllowVolumeExpansion:  True
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     WaitForFirstConsumer
Events:                <none>

We can see that “AllowVolumeExpansion” is “True” so we can proceed.

Growing the PersistentVolumeClaim

1. Get the name of the PVC using kubectl get pvc [-n <namespace] command. The output from above will be referenced again. We can see the PVC is named dremio-master-volume-dremio-master-0 from the output.
2. Edit the PVC definition with kubectl edit dremio-master-volume-dremio-master-0 [-n <namespace>] command. This will open an interactive editor to a YAML-formatted definition of the PVC. Navigate to the “spec” heading, locate the “storage” resource, and edit it to the desired size.

spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 721Gi <<< edit this value
  storageClassName: default
  volumeMode: Filesystem
  volumeName: pvc-b843efb1-0f6f-42ac-8195-9d3cb0e81974

3. Save and quit the editor. You should see the following indicating the change was saved:
   persistentvolumeclaim/dremio-master-volume-dremio-master-0 edited

Clearing the existing StatefulSet spec

1. Here we will delete the StatefulSet object with the older storage request, to allow creation of a new StatefulSet with the desired storage request. We will use the command kubectl delete sts --cascade=orphan dremio-master [-n <namespace>] and should see the following returned:

statefulset.apps "dremio-master" deleted

Use of the --cascade=orphan flag means the StatefulSet is deleted, but the pods launched by the StatefulSet are not deleted or restarted.

Applying the new StatefulSet spec

1. Edit the values.yaml for the deployment to set the coordinator “volumeSize” parameter to the desired size, save the change, and exit the editor.
2. Use helm upgrade <deployment name> ./values.yaml [-n <namespace>] to apply the new parameter. The StatefulSet will be recreated but no other changes should be made to the deployment from this step since all other values and specifications are unchanged. The new PVC size will not be reflected until the dremio-master-0 pod is restarted, however.

Restart the dremio-master-0

1. To allow the increased space to be used by the pod, scale the StatefulSet to 0 replicas with the following command:
   kubectl scale sts dremio-master --replicas=0 [-n <namespace>]
   You will see the dremio-master-0 pod be terminated by the Kubernetes control plane.
2. When this completes, scale the STS back to 1 replica with the command kubectl scale sts dremio-master --replicas=1 [-n <namespace>] and you will see the dremio-master-0 pod is created again.

Confirm the new space is usable

1. When the dremio-master-0 pod is ready, you can check the PVC size with kubectl get pvc [-n <namespace>] and you should see the updated size reflected.
2. You can then check the filesystem in the pod with the command kubectl exec dremio-master-0 -c dremio-master-coordinator -- df -h and you should see the filesystem has been grown to the new size.

devin [ ~/dremio-cloud-tools/charts/dremio_v2 ]$ kubectl exec dremio-master-0 -c dremio-master-coordinator -- df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay         124G   26G   99G  21% /
tmpfs            64M     0   64M   0% /dev
/dev/root       124G   26G   99G  21% /etc/hosts
shm              64M     0   64M   0% /dev/shm
/dev/sdc        709G   80M  709G   1% /opt/dremio/data
tmpfs            13G   12K   13G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs           7.9G     0  7.9G   0% /proc/acpi
tmpfs           7.9G     0  7.9G   0% /proc/scsi
tmpfs           7.9G     0  7.9G   0% /sys/firmware
devin [ ~/dremio-cloud-tools/charts/dremio_v2 ]$ 

Further Reading

https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/