How to collect thread and heap dumps from a Dremio process – Dremio Support

Overview

Sometimes it is necessary to collect further diagnostic information from a Dremio process to facilitate debugging by Dremio Support. This article outlines how to collect JVM thread and heap dump information

Applies To

All Dremio releases

Details

In this article we'll show some examples from a simple single line command to a script to collect this information.

Note: some tools like jstack require a JDK to be installed

The following commands will show $DREMIO_PID which should be replaced with the dremio process ID or alternatively the user can set the environment variable to the process id

DREMIO_PID=12345

Thread dumps

Thread dumps are useful to see what a process is doing at a certain point in time. One single thread dump is almost never sufficient enough, several are required to get a more holistic picture of the process over a period of time.

single line (outputs 60 files every 5 seconds)

for i in {1..60}; do jstack -l $DREMIO_PID > threadDump-$(date +%Y-%m-%d_%H_%M_%S).txt ; sleep 5 ; done

single line if jstack is not available

for i in {1..60}; do jcmd $DREMIO_PID Thread.print -l > threadDump-$(date +%Y-%m-%d_%H_%M_%S).txt ; sleep 5 ; done

script (generates multiple sequential files)

#!/bin/bash

for i in {1..60}
do
   jstack -l $DREMIO_PID > ThreadDump$i.txt
   sleep 5
done

Heap dumps

Heap dumps are commonly generated after a JVM process has run out of heap memory. As long as the process is run with the flags:

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<path>

It is also possible to manually generate a heap dump if required.

single line (path and filename can be adjusted to suit)

jmap -dump:live,file=/tmp/dremio-$(hostname)-heapdump.hprof $DREMIO_PID

Overview

Applies To

Details

Thread dumps

Heap dumps

Further Reading

Related articles