Overview
Query fails with StackOverflowError.
Applies To
Any Dremio release.
Details
In some cases, a query may include a long series or list of terms, such as in an “IN” clause with several hundred or more filter terms. This will cause the query to fail and the following message may be logged or found in the query profile from Dremio:
SYSTEM ERROR: StackOverflowError
(com.dremio.exec.work.foreman.ForemanException) Unexpected exception during fragment initialization: java.lang.StackOverflowError
com.dremio.exec.work.foreman.AttemptManager.run():439
com.dremio.context.RequestContext.run():95
com.dremio.common.concurrent.ContextMigratingExecutorService.lambda$decorate$3():199
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():750
Caused By (java.util.concurrent.ExecutionException) java.lang.StackOverflowError
Cause
This occurs when the combined size in bytes of all items in the list exceed the stack size for a given CPU thread in the Java Virtual Machine (JVM) running Dremio.
Workaround
As a workaround, it’s possible to add the following to the DREMIO_JAVA_SERVER_EXTRA_OPTS=
in the dremio-env
file: "-Xss=2048k"
. This will set the stack size to 2 MB. You may be able to complete the query with a lower value than 2048k, however, this property requires a restart to take effect.
Solution
While the workaround will allow most queries to complete with a long list of terms in a filter clause, some use cases will benefit from using a table to filter against, instead of a long list of items. If you need assistance developing an optimal access pattern for your use case, please contact your Dremio account team for details.