Reported Issue
A new install of Dremio on Kubernetes fails to start, failing with the message "Could not create the Java Virtual Machine".
Relevant Versions
Dremio v25.x and higher, running with Java 9 or later.
Troubleshooting Steps
Review the server.log or if running a Kubernetes deployment, run the command "kubectl logs dremio-master-0"
Here is an example error message:
dremio-executor starting dremio
dremio-executor [2024-06-06T13:23:03.388+0000][0.001s][error][logging] Error opening log file '/opt/dremio/data/logs/gc-2024-06-06_13-23-03.log': No such file or directory
dremio-executor [2024-06-06T13:23:03.388+0000][0.001s][error][logging] Initialization of output 'file=/opt/dremio/data/logs/gc-%t.log' using options 'filecount=1,filesize=4M' failed.
dremio-executor Invalid -Xlog option '-Xlog:gc*,classhisto*=trace:file=/opt/dremio/data/logs/gc-%t.log:uptime,time,tags,level:filecount=1,filesize=4M', see error log for details.
dremio-executor Error: Could not create the Java Virtual Machine.
dremio-executor Error: A fatal exception has occurred. Program will exit.
Stream closed EOF for enterprise-dremio-platform-dev/dremio-executor-0 (chown-cloudcache-directory)
Stream closed EOF for enterprise-dremio-platform-dev/dremio-executor-0 (dremio-executor)
Stream closed EOF for enterprise-dremio-platform-dev/dremio-executor-0 (wait-for-zookeeper)
Stream closed EOF for enterprise-dremio-platform-dev/dremio-executor-0
Cause
The issue occurs because the path to the GC log directory does not exist. Starting with Java 9, unrecognised or invalid JVM options cause the JVM to fail to start, whereas in Java 8, it would only trigger a warning. This change was introduced to make JVM behaviour more predictable and help administrators detect configuration issues early.
Steps to Resolve
1. Correct the GC logging path to a pre-existing directory in the `extraStartParams` section of the Helm chart's `values.yaml` file. Example for Dremio v25 and Java 11:
extraStartParams: >-
-Ddremio.log.path=/opt/dremio/data
-Xlog:gc*,classhisto*=trace:file=/opt/dremio/data/gc-%t.log:uptime,time,tags,level:filecount=5,filesize=4M
...
2. If you prefer the GC log files to remain inside the logs folder, follow the steps in option 1 to allow Dremio to create the missing directory during the initial startup. After a successful startup, Dremio will have created the missing directory "/opt/dremio/data/logs" on all Dremio pods due to the setting of "-Ddremio.log.path". You can now revert to your desired GC log path in "values.yaml", followed by an uninstall / install the helm chart once more, for the change to take effect.
Example for v25 and java 11
extraStartParams: >-
-Ddremio.log.path=/opt/dremio/data/logs
-Xlog:gc*,classhisto*=trace:file=/opt/dremio/data/gc-%t.log:uptime,time,tags,level:filecount=5,filesize=4M
...
Next Steps
Once the config changes have been applied restart Dremio. The startup may fail once again on a different parameter, review the log file and repeat the process.