Summary
Steps to gather extra logs, such as Thread Dumps and Heap Dumps, for Dremio processes.
Reported Issue
Thread dumps and heap dumps are often required in various scenarios, such as memory issues, hung jobs, slow executions, and more.
Overview
Sometimes it is necessary to collect further diagnostic information from a Dremio process to facilitate debugging by Dremio Support. This article outlines how to collect JVM thread and heap dump information
Note: this is a manual process and is superseded by using tools like DDC to collect this kind of diagnostic information. See https://support.dremio.com/hc/en-us/articles/15560006579739
Relevant Versions Tools and Integrations
All Dremio releases
Steps to Resolve
In this article we'll show some examples from a simple single line command to a script to collect this information.
Note: some tools like jstack require a JDK to be installed
The following commands will show $DREMIO_PID which should be replaced with the dremio process ID or alternatively the user can set the environment variable to the process id
DREMIO_PID=12345
Thread dumps
Thread dumps are useful to see what a process is doing at a certain point in time. One single thread dump is almost never sufficient enough, several are required to get a more holistic picture of the process over a period of time.
single line (outputs 60 files every 5 seconds)
for i in {1..60}; do jstack -l $DREMIO_PID > threadDump-$(date +%Y-%m-%d_%H_%M_%S).txt ; sleep 5 ; done
single line if jstack is not available
for i in {1..60}; do jcmd $DREMIO_PID Thread.print -l > threadDump-$(date +%Y-%m-%d_%H_%M_%S).txt ; sleep 5 ; done
script (generates multiple sequential files)
#!/bin/bash
for i in {1..60}
do
jstack -l $DREMIO_PID > ThreadDump$i.txt
sleep 5
done
Heap dumps
Heap dumps are commonly generated after a JVM process has run out of heap memory. As long as the process is run with the flags:
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<path>
It is also possible to manually generate a heap dump if required.
single line (path and filename can be adjusted to suit)
jmap -dump:file=/tmp/dremio-$(hostname)-heapdump.hprof $DREMIO_PID
Common Challenges
To identify changes, it's essential to capture the stack at different points in time. Therefore, collecting multiple dumps at regular intervals is necessary.
Additional Resources
Oracle java articles on jvm tools
https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr014.html
https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr016.html
https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr006.html