Overview

The disk will fill up on zookeeper pods as the logs and snapshots never purge.

Applies To

All versions of the helm chart that are still using the older k8s.gcr.io/kubernetes-zookeeper zookeeper image, which you can find out by running the following on your deployment (change -n to match the namespace where you have deployed dremio).

kubectl get pod -n default zk-0 -o yaml | grep image

Details

The docker version of zookeeper we were using had a typo for the purge interval, disabling purge permanently. Which you can see here or by running the following command with docker installed

dremio_v2 (master) ✗  docker run -it k8s.gcr.io/kubernetes-zookeeper:1.0-3.4.10 grep purge\.purge ./usr/bin/start-zookeeper 

echo "autopurge.purgeInteval=$PURGE_INTERVAL" >> $CONFIG_FILE

As you can see, Interval is misspelled as Inteval, and therefore, the configuration file is never correctly set.

Solution

To fix this, the easiest approach is to upgrade the helm chart from the source by running the following command where you checked the source out:

git pull origin master

Then update the values.yaml file under the zookeeper tag to look like the following:

# Zookeeper
zookeeper:
# The Zookeeper image used in the cluster.
  image:zookeeper
  imageTag: 3.8.0

Then run the following, changing my-release to match the release name of your release


helm upgrade -n default my-release .

After the upgrade has taken you should see the newer version of zookeeper on your pods

kubectl get pod -n default zk-0 -o yaml | grep image
image: zookeeper:3.8.0
imagePullPolicy: Always
image: docker.io/library/zookeeper:3.8.0

For those who cannot easily upgrade the helm chart

However, as many people have forked the helm chart, updating to the latest helm chart may be time-consuming. It is possible just to update just the YAML for Zookeeper.

Navigate to where you have the chart and cd into the templates folder

dremio-cloud-tools (master) ✗ cd charts/dremio_v2/templates

Then use curl to download a more recent version of the zookeeper yaml

curl -o zookeeper.yaml https://raw.githubusercontent.com/dremio/dremio-cloud-tools/master/charts/dremio_v2/templates/zookeeper.yaml

Now cd back up to where you have your helm chart checked out and run helm upgrade with the new zookeeper values passed in

cd ../
helm upgrade -n default my-release . --set zookeeper.image=zookeeper --set zookeeper.imageTag=3.8.0