How to collect thread and heap dumps from a Dremio process – Dremio Support

Overview

Sometimes it is necessary to collect further diagnostic information from a Dremio process to facilitate debugging by Dremio Support. This article outlines how to collect JVM thread and heap dump information

Note: this is a manual process and is superseded by using tools like DDC to collect this kind of diagnostic information. See https://support.dremio.com/hc/en-us/articles/15560006579739

Applies To

All Dremio releases

Details

In this article we'll show some examples from a simple single line command to a script to collect this information.

Note: some tools like jstack require a JDK to be installed

The following commands will show $DREMIO_PID which should be replaced with the dremio process ID or alternatively the user can set the environment variable to the process id

DREMIO_PID=12345

Thread dumps

Thread dumps are useful to see what a process is doing at a certain point in time. One single thread dump is almost never sufficient enough, several are required to get a more holistic picture of the process over a period of time.

single line (outputs 60 files every 5 seconds)

for i in {1..60}; do jstack -l $DREMIO_PID > threadDump-$(date +%Y-%m-%d_%H_%M_%S).txt ; sleep 5 ; done

single line if jstack is not available

for i in {1..60}; do jcmd $DREMIO_PID Thread.print -l > threadDump-$(date +%Y-%m-%d_%H_%M_%S).txt ; sleep 5 ; done

script (generates multiple sequential files)

#!/bin/bash

for i in {1..60}
do
   jstack -l $DREMIO_PID > ThreadDump$i.txt
   sleep 5
done

Heap dumps

Heap dumps are commonly generated after a JVM process has run out of heap memory. As long as the process is run with the flags:

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<path>

It is also possible to manually generate a heap dump if required.

single line (path and filename can be adjusted to suit)

jmap -dump:live,file=/tmp/dremio-$(hostname)-heapdump.hprof $DREMIO_PID

Overview

Applies To

Details

Thread dumps

Heap dumps

Further Reading

Related articles