Query fails with StackOverflowError.
Any Dremio release.
In some cases, a query may include a long series or list of terms, such as in an “IN” clause with several hundred or more filter terms. This will cause the query to fail and the following message may be logged or found in the query profile from Dremio:
SYSTEM ERROR: StackOverflowError (com.dremio.exec.work.foreman.ForemanException) Unexpected exception during fragment initialization: java.lang.StackOverflowError com.dremio.exec.work.foreman.AttemptManager.run():439 com.dremio.context.RequestContext.run():95 com.dremio.common.concurrent.ContextMigratingExecutorService.lambda$decorate$3():199 java.util.concurrent.ThreadPoolExecutor.runWorker():1149 java.util.concurrent.ThreadPoolExecutor$Worker.run():624 java.lang.Thread.run():750 Caused By (java.util.concurrent.ExecutionException) java.lang.StackOverflowError
This occurs when the combined size in bytes of all items in the list exceed the stack size for a given CPU thread in the Java Virtual Machine (JVM) running Dremio.
As a workaround, it’s possible to add the following to the
DREMIO_JAVA_SERVER_EXTRA_OPTS= in the
"-Xss=2048k". This will set the stack size to 2 MB. You may be able to complete the query with a lower value than 2048k, however, this property requires a restart to take effect.
While the workaround will allow most queries to complete with a long list of terms in a filter clause, some use cases will benefit from using a table to filter against, instead of a long list of items. If you need assistance developing an optimal access pattern for your use case, please contact your Dremio account team for details.