Overview
Sometimes it is necessary to collect further diagnostic information from a Dremio process to facilitate debugging by Dremio Support. This article outlines how to collect JVM thread and heap dump information
Applies To
All Dremio releases
Details
In this article we'll show some examples from a simple single line command to a script to collect this information.
Note: some tools like jstack require a JDK to be installed
The following commands will show $DREMIO_PID which should be replaced with the dremio process ID or alternatively the user can set the environment variable to the process id
DREMIO_PID=12345
Thread dumps
Thread dumps are useful to see what a process is doing at a certain point in time. One single thread dump is almost never sufficient enough, several are required to get a more holistic picture of the process over a period of time.
single line (outputs 60 files every 5 seconds)
for i in {1..60}; do jstack -l $DREMIO_PID > threadDump-$(date +%Y-%m-%d_%H_%M_%S).txt ; sleep 5 ; done
single line if jstack is not available
for i in {1..60}; do jcmd $DREMIO_PID Thread.print -l > threadDump-$(date +%Y-%m-%d_%H_%M_%S).txt ; sleep 5 ; done
script (generates multiple sequential files)
#!/bin/bash
for i in {1..60}
do
jstack -l $DREMIO_PID > ThreadDump$i.txt
sleep 5
done
Heap dumps
Heap dumps are commonly generated after a JVM process has run out of heap memory. As long as the process is run with the flags:
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<path>
It is also possible to manually generate a heap dump if required.
single line (path and filename can be adjusted to suit)
jmap -dump:live,file=/tmp/dremio-$(hostname)-heapdump.hprof $DREMIO_PID
Further Reading
Oracle java articles on jvm tools
https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr014.html
https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr016.html
https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr006.html