Summary
This article answers the question in regards to how many concurrent queries a Dremio cluster can handle.
Reported Issue
When planning a Dremio cluster, the user may have a specific request in regards to how many concurrent queries are needed to be supported.
Overview
In regards to the maximum concurrent number of queries, Dremio doesn't have such a concept, as not all queries are the same and generally they vary a lot based on their cost to run.
Relevant Versions, Tools & Integrations
This applies to all Dremio releases.
Steps to Resolve
There are many factors involved when deciding how many queries a cluster can process concomitantly, from the size of the cluster, to the type of data sources, how the cache is being configured, the cost of the queries, locking/data consistency requirements of the queries, whether or not reflections are being leveraged, to the type of drivers/clients, etc.
Ideally, once a certain number of concurrent queries run and you start to see failures, the next step is to identify the bottleneck and decide how to fix that. Here, making sure that Workload Management is tuned properly is very important, as certain rules/queues/engines can be tweaked to run workloads to the proper executors. Generally, if the bottleneck is somewhere with the executors, then more executors can be added or more resources can be added to the existing executors. If the bottleneck is with the master, then scale out coordinators can be added to allow the master to process more queries. Master is generally bounded by the heap size and number of CPUs. Once you run with 32-48 vCPUs and 31GB heap on the master, then adding scale out coordinators is the next to be considered.
To give a ballpark estimation though, we have users who deal with concurrent queries in the number of thousands at the time. Please see Lenoy's talk here on what he does to reach those numbers: https://www.youtube.com/watch?v=mwF_98WKGHs
Tips & Tricks / Best Practices / Recommendations / FAQ
In the Dremio Limits documentation, one can read that the "Total active queries (queued and executing)" is configured to 1,000. That's just a guardrail, so Dremio administrators get alerted when such high usage happens, sometimes there could be bottlenecks in the system, thus the queries get queued, which administrators want to know about. Also, it applies to the UI ran queries. One can change that limit by using support key "coordinator.alive_queries.limit" -- one can increase it to a higher value in case the limit is reached and the Dremio cluster performs well.
Additional Resources
Concurrency in Iceberg Tables
Concurrency
Concurrency Control
Dremio Load Testing