Overview
If the Dremio JVM memory is sized too small the process may not be able to create any more threads and will exit
Applies To
All versions
Details
The user will noticed the process exiting and possibly restart with the following error being observed in the server.log
2021-12-20 09:00:47,307 [1e3fb9bf-f6cd-811c-3fe0-522b696a7300:foreman] ERROR ROOT - Dremio is exiting. There was insufficient heap memory to continue operating.
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
at com.dremio.common.concurrent.ContextMigratingExecutorService.execute(ContextMigratingExecutorService.java:101)
at com.dremio.common.SerializedExecutor.execute(SerializedExecutor.java:129)
at com.dremio.exec.planner.observer.OutOfBandAttemptObserver.execute(OutOfBandAttemptObserver.java:278)
at com.dremio.exec.planner.observer.OutOfBandAttemptObserver.beginState(OutOfBandAttemptObserver.java:70)
at com.dremio.exec.planner.observer.DelegatingAttemptObserver.beginState(DelegatingAttemptObserver.java:55)
at com.dremio.exec.planner.observer.AttemptObservers.beginState(AttemptObservers.java:62)
at com.dremio.exec.work.foreman.AttemptManager.run(AttemptManager.java:341)
at com.dremio.context.RequestContext.run(RequestContext.java:95)
at com.dremio.common.concurrent.ContextMigratingExecutorService.lambda$decorate$3(ContextMigratingExecutorService.java:199)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)Cause
The Dremio JVM process will attempt to create new thread as it needs them. If the process is creating too many threads, there is not enough memory available for the operating system to create more threads or the system limits for thread creation are too low, this error can occur.
Solution
Check your process limits for the user
ulimit -u
Or process limits for the running process
$ cat /proc/1396/limits
Limit Soft Limit Hard Limit Units
...
Max processes 59462 59462 processes
...
Depending on your install type the limits may inherit from the user or a system service.
Check that you have not inadvertently sized the JVM heap too large that the remaining memory for the OS is too small
If you still keep observing this error after increasing limits then contact Dremio Support for assistance.
Further Reading
Dremio docs for Memory settings - https://docs.dremio.com/deployment/memory-config/
Background on native thread OOM - https://dzone.com/articles/troubleshoot-outofmemoryerror-unable-to-create-new