Summary
This article discusses how to resolve the Error while applying rule AggregateRule, args [rel#371763991:LogicalAggregate.NONE.ANY([]).[](input=RelSubset#371763990,group={},agg#0=COUNT()), rel#371763989:LogicalUnion.NONE.ANY([]).[](input#0=RelSubset#371763983,input#1=RelSubset#371763988,all=true)] error when running VDS queries.
The problem can often be temporarily resolved by unformatting and re-promoting the underlying parquet file(s).
Reported Issue
When attempting to run VDS queries, these would fail in Dremio with the below error.
Error while applying rule AggregateRule, args [rel#371763991:LogicalAggregate.NONE.ANY([]).[](input=RelSubset#371763990,group={},agg#0=COUNT()), rel#371763989:LogicalUnion.NONE.ANY([]).[](input#0=RelSubset#371763983,input#1=RelSubset#371763988,all=true)]
However unformatting and re-promoting the underlying parquet file(s) would allow the query to temporarily work.
Relevant Versions
This issue applies to Dremio versions 24.3.x and on wards.
Troubleshooting Steps
1. Navigate to the failing job profile as per the following documentation link steps: https://docs.dremio.com/current/sonar/monitoring/jobs/raw-profile/
2. After navigating to the "Raw Profile", click on the "Error" tab of the profile.
3. Confirm that the error is the same as mentioned in this article. As an example, the following error can be seen in the "Verbose" section of the "Error" tab of the job profile:
PLAN ERROR: Error while applying rule AggregateRule, args [rel#371763991:LogicalAggregate.NONE.ANY([]).[](input=RelSubset#371763990,group={},agg#0=COUNT()), rel#371763989:LogicalUnion.NONE.ANY([]).[](input#0=RelSubset#371763983,input#1=RelSubset#371763988,all=true)]
SQL Query SELECT DISTINCT(deviceId) FROM "myspace".myfolder."my_vds" WHERE "my_vds".deviceId IN('ABC123')
(java.lang.RuntimeException) Error while applying rule AggregateRule, args [rel#371763991:LogicalAggregate.NONE.ANY([]).[](input=RelSubset#371763990,group={},agg#0=COUNT()), rel#371763989:LogicalUnion.NONE.ANY([]).[](input#0=RelSubset#371763983,input#1=RelSubset#371763988,all=true)]
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch():243
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp():644
com.dremio.exec.planner.DremioVolcanoPlanner.findBestExp():113
org.apache.calcite.tools.Programs$RuleSetProgram.run():321
com.dremio.exec.planner.sql.handlers.PlannerUtil.lambda$transform$1():180
com.dremio.exec.planner.sql.handlers.PlannerUtil.doTransform():204
com.dremio.exec.planner.sql.handlers.PlannerUtil.transform():193
com.dremio.exec.planner.sql.handlers.DrelTransformer.convertToDrel():142
com.dremio.exec.planner.sql.handlers.DrelTransformer.convertToDrel():100
com.dremio.exec.planner.sql.handlers.query.NormalHandler.getPlan():87
com.dremio.exec.planner.sql.handlers.commands.HandlerToExec.plan():59
com.dremio.exec.work.foreman.AttemptManager.plan():561
com.dremio.exec.work.foreman.AttemptManager.lambda$run$4():458
com.dremio.service.commandpool.ReleasableBoundCommandPool.lambda$getWrappedCommand$3():140
com.dremio.service.commandpool.CommandWrapper.run():70
com.dremio.context.RequestContext.run():109
com.dremio.common.concurrent.ContextMigratingExecutorService.lambda$decorate$4():227
com.dremio.common.concurrent.ContextMigratingExecutorService$ComparableRunnable.run():207
java.util.concurrent.Executors$RunnableAdapter.call():511
java.util.concurrent.FutureTask.run():266
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():750
Caused By (java.lang.RuntimeException) Error occurred while applying rule AggregateRule
org.apache.calcite.plan.volcano.VolcanoRuleCall.transformTo():151
org.apache.calcite.plan.RelOptRuleCall.transformTo():250
org.apache.calcite.plan.RelOptRuleCall.transformTo():265
com.dremio.exec.planner.logical.AggregateRule.onMatch():56
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch():216
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp():644
com.dremio.exec.planner.DremioVolcanoPlanner.findBestExp():113
org.apache.calcite.tools.Programs$RuleSetProgram.run():321
com.dremio.exec.planner.sql.handlers.PlannerUtil.lambda$transform$1():180
com.dremio.exec.planner.sql.handlers.PlannerUtil.doTransform():204
com.dremio.exec.planner.sql.handlers.PlannerUtil.transform():193
com.dremio.exec.planner.sql.handlers.DrelTransformer.convertToDrel():142
com.dremio.exec.planner.sql.handlers.DrelTransformer.convertToDrel():100
com.dremio.exec.planner.sql.handlers.query.NormalHandler.getPlan():87
com.dremio.exec.planner.sql.handlers.commands.HandlerToExec.plan():59
com.dremio.exec.work.foreman.AttemptManager.plan():561
com.dremio.exec.work.foreman.AttemptManager.lambda$run$4():458
com.dremio.service.commandpool.ReleasableBoundCommandPool.lambda$getWrappedCommand$3():140
com.dremio.service.commandpool.CommandWrapper.run():70
com.dremio.context.RequestContext.run():109
com.dremio.common.concurrent.ContextMigratingExecutorService.lambda$decorate$4():227
com.dremio.common.concurrent.ContextMigratingExecutorService$ComparableRunnable.run():207
java.util.concurrent.Executors$RunnableAdapter.call():511
java.util.concurrent.FutureTask.run():266
java.util.concurrent.ThreadPoolExecutor.runWorker():1149
java.util.concurrent.ThreadPoolExecutor$Worker.run():624
java.lang.Thread.run():750
Caused By (java.lang.NullPointerException) null
4. Carry out the "Steps to Resolve" described below.
Cause
This was an issue in the Dremio planner when running queries against VDS that made use of the UNION function.
Steps to Resolve
Upgrade to Dremio v25.2.2 or higher.
Additional Resources
Viewing a Raw Profile: https://docs.dremio.com/current/sonar/monitoring/jobs/raw-profile/