Summary
This article details the steps to increase stack size for Java running Dremio.
Reported Issue
In some cases, a query may include a long series or list of terms, such as in an "IN" clause with several hundred or more filter terms. This will cause the query to fail and the following message may be logged or found in the query profile from Dremio:
SYSTEM ERROR: StackOverflowError (com.dremio.exec.work.foreman.ForemanException) Unexpected exception during fragment initialization: java.lang.StackOverflowError com.dremio.exec.work.foreman.AttemptManager.run():439 com.dremio.context.RequestContext.run():95 com.dremio.common.concurrent.ContextMigratingExecutorService.lambda$decorate$3():199 java.util.concurrent.ThreadPoolExecutor.runWorker():1149 java.util.concurrent.ThreadPoolExecutor$Worker.run():624 java.lang.Thread.run():750 Caused By (java.util.concurrent.ExecutionException) java.lang.StackOverflowError
Relevant Versions
Any Dremio release.
Steps to Resolve
To increase the stack size add the following to the DREMIO_JAVA_SERVER_EXTRA_OPTS=
line in the dremio-env
file: "-Xss=2048k"
. This requires a restart to take effect.
Common Challenges
Some use cases still may not see the query succeed and will benefit from using a table to filter against, instead of a long list of items.