Summary
This article outlines how to collect a jstack output within your K8's pod.
Reported Issue
Any performance issue or coordinator down, blocked upon restart, etc...
Overview
The steps below outlines a simple way to take thread dumps within your master pod or coordinator.
Note: this is a manual process and is superseded by using tools like DDC to collect this kind of diagnostic information. See https://support.dremio.com/hc/en-us/articles/15560006579739
Relevant Versions Tools and Integrations
All Dremio releases
Steps to Resolve
- Logon to the dremio pod with command
kubectl -it exec dremio-master-0 bash`
- Make sure you are logged in as the unix id running the Dremio process and run command
mkdir /tmp/jstack
- Create a file called jstack.sh with content below:
#!/bin/bash
which you will copy into the pod's jstack directory ( you will see that the pid is equal to 1. It usually is in the dremio master pod).
for i in `seq -w 3 1 300`
do
jstack -l 1 > ThreadDump$i.txt
sleep 1
done - You will need to exit the pod before you run command:
kubectl cp /Dir_of_your_choice/jstack.sh dremio-master-0:/tmp/jstack/jstack.sh
- Back in the pod, inside /tmp/jstack , run:
chmod +x jstack.sh
- then run:
/jstack.sh
- While it's running, reproduce the problem you are investigating in Dremio.
- Stop jstack.sh. You will find a few .txt files inside the /tmp/jstack directory.
- Run commands:
tar -cvf jstack.tar *.txt
gzip jstack.tar - Exit the pod then run the command:
kubectl cp dremio-master-0:/tmp/jstack/jstack.tar.gz /Dir_of_your_choice/jstack.tar.gz
- Attach file jstack.tar.gz to the Zendesk ticket
Tips and Tricks
You can also use this one-liner instead of creating the script in step 3:
for i in {1..5}; do jstack -l $DREMIO_PID > threadDump-$(date +%Y-%m-%d_%H_%M_%S).txt ; sleep 5 ; done
Common Challenges
Additional Resources
How to collect thread and heap dumps from a Dremio process – Dremio Support