Summary
In a Kubernetes deployment of Dremio, expanding the /opt/dremio/data volume for the dremio-master-0 pod requires additional steps due to limitations with Kubernetes StatefulSets. This article details the process for increasing the storage volume size.
Reported Issue
Insufficient space messages from Dremio logs, or a choice to expand the volume in place.
Overview
In a Dremio deployment on Kubernetes, it requires extra handling to grow the /opt/dremio/data
mount point which the dremio-master-0 depends upon. This article will detail the process of increasing space for the volume in the deployment.
Relevant Versions Tools and Integrations
Any Kubernetes-based deployment of Dremio.
Steps to Resolve
Prequisite
Ensure the PVC/PV "StorageClass" supports "AllowVolumeExpansion" and it is set to "True."
Growing the PersistentVolumeClaim
- Get the name of the PVC using
kubectl get pvc [-n <namespace]
. - Edit the PVC definition with
kubectl edit pvc [name] [-n <namespace>]
and update the storage size. - Save and exit the editor.
Clearing the existing StatefulSet spec
- Delete the StatefulSet with
kubectl delete sts --cascade=orphan dremio-master [-n <namespace>]
.
Applying the new StatefulSet spec
- Edit values.yaml to set the coordinator volumeSize parameter.
- Apply with
helm upgrade <name> ./values.yaml [-n <namespace>]
.
Restart the dremio-master-0
- Scale the STS to 0 replicas:
kubectl scale sts dremio-master --replicas=0
- Then scale back to 1 replica:
kubectl scale sts dremio-master --replicas=1
Confirm the new space is usable
- Check PVC size with
kubectl get pvc
- Check filesystem in pod with
kubectl exec dremio-master-0 -c dremio-master-coordinator -- df -h
Common Challenges
This does require a downtime, so plan as needed for an interruption of Dremio services.
Additional Resources
https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/