Overview
Using queues to handle the flow of various requests in Dremio is important to keeping various workloads running smoothly. A queue in Dremio is a list of queries with one or more properties in common, identified by rules set by cluster administrators. Queues can have specific behaviors configured, including time limits, memory limits, and routing to specific groups of nodes labeled as "engines." This article will share some best practices and anti-patterns for using queues in Dremio.
Relevant Versions Tools and Integrations
All currently supported Dremio Enterprise releases.
Configuring Queues
Configuring a queue is done through the "Settings" panel under the Queues heading, as seen below:
To configure a queue, hover over the queue row and click on the Edit (pencil) icon which will appear on the right side of the row to open a dialog about the queue name. Here is an example of the "UI Previews" queue pre-configured in Dremio:
Note the various fields including concurrency, time limits, and more. A complete guide to queue features can be found in the Workload Management documentation.
Configuring Engines
Using queues to send jobs to an "engine" is an important part of deploying a high-performance Dremio cluster. An engine is a named group of executors as shown below:
In a queue configuration, selecting an engine will ensure all queries from that queue are only processed on executors grouped into that engine. This makes it possible to dedicate some executors to metadata refresh jobs, costly reflection jobs, or specific hardware as application workloads demand.
The below example shows "High-Cost Reflections" routing all queries that match the rule set(s) placing queries on that queue, to the executor nodes in the "example2" engine:
An important detail about Engine assignment!
When using engines with queues, it is important to ensure that every executor is placed in some engine, with a "default" engine being configured if required, and that no queues route the queries in the queue to the "Any" Engine Name option, unless the queries in that queue will actually be allowed to run on any executors.
This is because any queue using "Any" will assign jobs to any executor in the cluster, including executors in engines that are planned as dedicated for specific jobs, which can disrupt performance.
In the "UI Previews" queue example shown again below, the "Any" selection indicates that the queries in the queue can be run on executors in the example1 and example2 engines. So, if administrators wanted to ensure that "UI Previews" queries are not placed in the example2 engine used for High Cost Reflections above, the "Any" option should not be used! Administrators should select "example1" instead, to route all "UI Previews" queries away from the example2 engine, and only to the executors in the example1 engine.