Overview

Using queues to handle the flow of various requests in Dremio is important to keeping various workloads running smoothly.
A queue in Dremio is a list of queries with one or more properties in common, identified by rules set by cluster administrators. Queues can have specific behaviors configured, including time limits, memory limits, and routing to specific groups of nodes labeled as “engines.” This article will share some best practices and anti-patterns for using queues in Dremio.

Applies To

All currently supported Dremio Enterprise releases.

Details

Configuring a queue is done through the “Settings” panel under the Queues heading, as seen below:

To configure a queue, hover over the queue row and click on the Edit (pencil) icon which will appear on the right side of the row to open a dialog about the queue name. Here is an example of the “UI Previews” queue pre-configured in Dremio:

Note the various fields including concurrency, time limits, and more. A complete guide to queue features can be found in the Workload Management documentation.

Suggested queues to add

While several useful queues are configured on a new Dremio installation, some queues can be added as usage increases to improve cluster performance, isolate jobs, or balance the usage of cluster resources.

Metadata refresh queue

A dedicated queue for metadata refresh jobs can be used to assign all such jobs to specific nodes by assigning an “engine” for the queue. This can be important in busy clusters using Iceberg or Delta Lake, where metadata refresh jobs are run in executor nodes and not on coordinators.

JDBC/ODBC “Express” queue

Using the ROUTING_QUEUE=<queue name> token in your JDBC or ODBC connect string will let you bypass all Workload Manager rules and place queries from that session directly on the named queue. If your application should always be processed on certain executors regardless of query cost or any other rules that may otherwise affect queue assignment, this approach will avoid any risk of those queries being placed in different executors.

Using Engines and Queues together

Using queues to send jobs to an “engine” is an important part of deploying a high-performance Dremio cluster. An engine is a named group of executors as shown below:

In a queue configuration, selecting an engine will ensure all queries from that queue are only processed on executors grouped into that engine. This makes it possible to dedicate some executors to metadata refresh jobs, costly reflection jobs, or specific hardware as application workloads demand.

The below example shows “High-Cost Reflections” routing all queries that match the rule set(s) placing queries on that queue, to the executor nodes in the “example2” engine:

An important detail about Engine assignment!

When using engines with queues, it is important to ensure that every executor is placed in some engine, with a “default” engine being configured if required, and that no queues route the queries in the queue to the “Any” Engine Name option, unless the queries in that queue will actually be allowed to run on any executors.

This is because any queue using “Any” will assign jobs to any executor in the cluster, including executors in engines that are planned as dedicated for specific jobs, which can disrupt performance.

In the “UI Previews” queue example shown again below, the “Any” selection indicates that the queries in the queue can be run on executors in the example1 and example2 engines. So, if administrators wanted to ensure that “UI Previews” queries are not placed in the example2 engine used for High Cost Reflections above, the “Any” option should not be used! Administrators should select “example1” instead, to route all “UI Previews” queries away from the example2 engine, and only to the executors in the example1 engine.