Overview
Using workload ‘engines’ to process some queries and jobs in specific Dremio executors is an important aspect of managing a Dremio deployment. An engine is a named group of machines or pods that can be assigned as a target for one or more queues in your Workload Manager. The type of infrastructure hosting the deployment will affect the way the cluster administrators can customize their deployments to define engines.
Applies To
All currently supported versions of Dremio Enterprise Edition installed on self-managed Linux hosts including cloud VM and on-premise servers.
Deploying a named group of executor nodes
Any executor node in an on-premise deployment is configured via the local dremio.conf
file and in this file, it’s possible to set a parameter key of services.node-tag:
with a string value, e.g.
services: {
executor.enabled: true,
node-tag: example1
}
Results
When the executor is launched, it will include this property when it registers with Zookeeper, and the value will be shown on the “Node Activity” page in the browser interface in the Settings tab for your Dremio.
For example:
Recommended Best Practice
When using named engines to route specific jobs to specific groups of nodes, it is a best practice to make sure that every executor is grouped into either the specific engine it is planned for or a “default” engine name, which can be any string, such as default
or main
to set all nodes into some engine. Cluster managers should not leave any nodes without a node-tag to avoid problems with queue assignment when configuring queues.