Overview
When specifying JVM options for Dremio in the dremio-env or in the Helm template values.yaml. It is important to specify the G1GC Garbage Collector as sometimes Dremio may not select a collector which could cause unexpected behaviour.
Applies To
All releases
Details
Usually by default Dremio will configure the G1GC collector. However if a user uses other JVM options such as -XX:+UseGCLogFileRotation then the JVM could end up with the default collector which on Java 8 for example is the ParallelGC collector.
For example; setting this in the helm template:
extraStartParams: >- -Ddremio.log.path=/opt/dremio/data/logs -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=4000k -XX:+UseGCLogFileRotation -Xloggc:/opt/dremio/data/log/gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintClassHistogramBeforeFullGC -XX:+PrintClassHistogramAfterFullGC
Will result in these JVM options
kubectl exec -c dremio-executor dremio-executor-0 -- bash -c 'jcmd 1 VM.flags | tr " " "\n"' 1: -XX:CICompilerCount=2 -XX:GCLogFileSize=4096000 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/dremio -XX:InitialHeapSize=251658240 -XX:MaxDirectMemorySize=2147483648 -XX:MaxHeapSize=4294967296 -XX:MaxNewSize=1431306240 -XX:MinHeapDeltaBytes=524288 -XX:NewSize=83886080 -XX:NumberOfGCLogFiles=5 -XX:OldSize=167772160 -XX:+PrintClassHistogramAfterFullGC -XX:+PrintClassHistogramBeforeFullGC -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseGCLogFileRotation -XX:+UseParallelGC
The same applies to all deployment types
You may observe a log output on first startup like so:
% kubectl logs dremio-master-0 | head -10 ####################################################################### WARNING: The default GC option (-XX:+UseG1GC) is conflicting with existing options DREMIO_JAVA_SERVER_EXTRA_OPTS (-Ddremio.log.path=/opt/dremio/data/log -XX:+AlwaysPreTouch -Xms1g -Xmx1g -Ddebug.allowTestApis=true -Xloggc:/opt/dremio/data/gc.log -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=4000k -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintClassHistogramBeforeFullGC -XX:+PrintClassHistogramAfterFullGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/dremio/data -XX:G1HeapRegionSize=32M -XX:MaxGCPauseMillis=500 -XX:InitiatingHeapOccupancyPercent=25 -XX:+PrintAdaptiveSizePolicy -XX:+PrintReferenceGC -XX:ErrorFile=/opt/dremio/data/hs_err_pid%p.log -XX:+UseCompressedClassPointers -XX:+UseGCLogFileRotation -Dzookeeper=zk-hs:2181 -Dservices.coordinator.enabled=true -Dservices.coordinator.master.enabled=true -Dservices.coordinator.master.embedded-zookeeper.enabled=false -Dservices.executor.enabled=false -Dservices.conduit.port=45679) and/or DREMIO_JAVA_EXTRA_OPTS (). So the default GC option (-XX:+UseG1GC) is not used. #######################################################################
You may also see "Full GC (Ergonomics)" in the GC logs as an indication that ParallelGC has been enabled.
Cause
The issue is caused by a regex check to make sure that the G1GC is does not conflict with any existing options. However this check can sometimes produce a false positive e.g. -XX:+UseGCLogFileRotation
This issue is currently being tracked in internal jira DX-54165. It has been fixed in versions 23.0.0 and later, and was ported to 21.7.0, and 22.1.5.
Workaround
The issue is easily avoided by always making sure you specify -XX:+UseG1GC alongside any other JVM options you wish to use. Note it might appear twice on the command line in a ps output but this is purely cosmetic.