Summary
On Kubernetes deployments ZooKeeper liveness/ readiness probes can fail, causing dremio-master-0 pod to restart.
Reported Issue
During normal operation of a Dremio cluster hosted on Kubernetes, we frequently see issues with customer clusters where the master pod will restart. This is normally due to an overloaded co-ordinator, where resource is insufficient for concurrent workload submitted.
However, this symptom can also be triggered by the failure of the Zookeeper liveness/ readiness probe. Netcat is configured to run the local calls to ensure the ZooKeeper pods are available, and report if the process has failed.
However, a bug has been identified where when a TCP RESET packet is returned from the probe, netcat will read this as a failed packet and mark the master pod as down, triggering a restart of the master pod.
Relevant Versions
All Kubernetes Dremio clusters where dremio master pods restart when not under load.
Cause
A bug has been identified where when a TCP RESET packet is returned from the probe, netcat will read this as a failed packet and mark the master pod as down, triggering a restart of the master pod.
Steps to Resolve
Note that remedial steps here are provided for the dremio_v2 charts.
To work around this behaviour, the ZooKeeper probes should be updated in the helm chart templates, and the changes pushed out via helm upgrade.
1. Correct the readiness probe. Edit zookeeper.yaml in the template location here for your chart, and update command:
command: ["/bin/bash", "-c", "[ \"$(echo ruok | nc 127.0.0.1 2181)\" == \"imok\" ]" ]
....so it reads:
command: ["/bin/bash", "-c", "[ \"$(echo ruok | (exec 3<>/dev/tcp/127.0.0.1/2181; cat >&3; cat <&3; exec 3<&-))\" == \"imok\" ]" ]
2. Repeat these steps for the liveness probe. Edit zookeeper.yaml in the template location here, and update command:
command: ["/bin/bash", "-c", "[ \"$(echo ruok | nc 127.0.0.1 2181)\" == \"imok\" ]" ]
....so it reads:
command: ["/bin/bash", "-c", "[ \"$(echo ruok | (exec 3<>/dev/tcp/127.0.0.1/2181; cat >&3; cat <&3; exec 3<&-))\" == \"imok\" ]" ]
Write the changes and exit.
3. Push the changes out to the ZooKeeper pods using helm:
helm upgrade -n <namespace> <cluster name> dremio_v2 -f dremio_v2/values.local.yaml
Be aware this will also trigger a master pod restart while ZooKeeper is down, so make the changes at a quiet time.