Summary
This article discusses how to resolve the IllegalStateException: Memory was leaked by query. Memory leaked error when running PowerBI queries that are accelerated by reflections in Dremio. NOTE: This issue is only relevant to customer environments using a scale-out coordinator as part of the deployment.
Reported Issue
When attempting to run queries that used reflections through PowerBI, these would fail with the below error due to the query being sent to the scale out coordinator however the query would technically complete within all phase states, with these being marked as FINISHED.
IllegalStateException: Memory was leaked by query. Memory leaked
However running the same query in the Dremio UI would succeed as the query in this case was being served by the master coordinator.
Relevant Versions
This issue applies to Dremio versions 25.2.2 and on wards.
Troubleshooting Steps
1. Navigate to the failing job profile as per the following documentation link steps: https://docs.dremio.com/current/sonar/monitoring/jobs/raw-profile/
2. After navigating to the "Raw Profile", under the Job Summary, verify that the query was run at the scale out coordinator per below example where the master is represented as dremio-master-0.dremio-cluster-pod.default.svc.cluster.local and the scale out as dremio-coordinator-0.dremio-cluster-pod.default.svc.cluster.local..
Job Summary
State: FAILED
Coordinator: dremio-coordinator-0.dremio-cluster-pod.default.svc.cluster.local
3. Still within the "Raw Profile", click on the "Acceleration" tab of the profile.
4. Confirm that the missing dataset ("unknown") is seen under the Substitution Info section per below example:
missing dataset ("unknown")
Raw Reflection (Type: raw, Refection Id: 26450972-7d99-444b-9e31-ba4eb65e915b): considered, matched, chosen.
5. Still within the "Raw Profile", click on the "Error" tab of the profile.
6. Confirm that the error is the same as mentioned in this article noting particularly the below p-s-r allocator being present in the stack. As an example, the following error can be seen in the "Verbose" section of the "Error" tab of the job profile:
SYSTEM ERROR: IllegalStateException: Memory was leaked by query. Memory leaked: (8192)
Outstanding child allocators :
Allocator(p-s-r-0559bc74-f6ea-4bf5-8909-02118cb72122) 0/8192/8192/9223372036854775807 (res/actual/peak/limit)
Allocator(query-planning:184b8296-57eb-6ccc-56c4-8872fb87af00) 0/8192/13568/268435456 (res/actual/peak/limit)
(java.lang.IllegalStateException) Memory was leaked by query. Memory leaked: (8192)
Outstanding child allocators :
Allocator(p-s-r-0559bc74-f6ea-4bf5-8909-02118cb72122) 0/8192/8192/9223372036854775807 (res/actual/peak/limit)
Allocator(query-planning:184b8296-57eb-6ccc-56c4-8872fb87af00) 0/8192/13568/268435456 (res/actual/peak/limit)
org.apache.arrow.memory.BaseAllocator.close():491
com.dremio.common.AutoCloseables.close():142
com.dremio.exec.ops.QueryContext.close():559
com.dremio.exec.ops.EnterpriseQueryContext.close():143
com.dremio.exec.work.foreman.AttemptManager$AttemptResult.suppressingClose():976
com.dremio.exec.work.foreman.AttemptManager$AttemptResult.close():1033
com.dremio.exec.work.foreman.AttemptManager.moveToState():1305
com.dremio.exec.work.foreman.AttemptManager$StateSwitch.processEvent():1381
com.dremio.exec.work.foreman.AttemptManager$StateSwitch.processEvent():1373
com.dremio.common.EventProcessor.processEvents():101
com.dremio.common.EventProcessor.sendEvent():61
com.dremio.exec.work.foreman.AttemptManager$StateSwitch.addEvent():1376
com.dremio.exec.work.foreman.AttemptManager.addToEventQueue():1386
com.dremio.exec.work.foreman.AttemptManager$CompletionListenerImpl.succeeded():318
com.dremio.exec.maestro.FragmentTracker.checkAndNotifyCompletionListener():347
com.dremio.exec.maestro.FragmentTracker.markNodeDone():268
com.dremio.exec.maestro.FragmentTracker.nodeCompleted():170
com.dremio.exec.maestro.QueryTrackerImpl.nodeCompleted():286
com.dremio.exec.maestro.MaestroServiceImpl$ExecToCoordStatusHandlerImpl.nodeQueryCompleted():362
com.dremio.sabot.rpc.CoordExecService$CoordExecProtocol.lambda$handle$1():309
java.util.concurrent.Executors$RunnableAdapter.call():515
java.util.concurrent.FutureTask.run():264
java.util.concurrent.ThreadPoolExecutor.runWorker():1128
java.util.concurrent.ThreadPoolExecutor$Worker.run():628
java.lang.Thread.run():829
4. Carry out the "Steps to Resolve" described below.
Cause
Queries that used reflections would fail through PowerBI with the above error stack (p-s-r) allocator indicating Dremio was trying to get the partition stats for the materialization of a missing reflection because of the metadata in the materialization cache of the scale out coordinator being in an inconsistent state.
Steps to Resolve
Upgrade to Dremio v25.2.4 or higher.
Additional Resources
Viewing a Raw Profile: https://docs.dremio.com/current/sonar/monitoring/jobs/raw-profile/
v25.2.4 Release Notes: https://docs.dremio.com/current/release-notes/version-250-release#2524-march-2025