Overview
Using workload "engines" to process execute particular queries and jobs in specific Dremio executors is an important aspect of managing a Dremio deployment. An engine is a named group of servers or pods that can be assigned as a target for one or more queues in your Workload Manager. This article will provide details and examples on deploying with different engines in a Kubernetes deployment.
Relevant Versions Tools and Integrations
All currently supported versions of Dremio Enterprise Edition launched with Dremio V2 Helm charts, found at: https://github.com/dremio/dremio-cloud-tools/tree/master/charts/dremio_v2
Steps to Resolve
Deploying Multiple Standardized Engines
# Engines
# Engine names be 47 characters or less and be lowercase alphanumber characters or '-'.
# Note: The number of executor pods will be the length of the array below * count.
engines: ["default"]
count: 3
# Executor volume size.
volumeSize: 128Gi
# ... (Additional configuration options omitted for brevity)
To change the name or number of engines and pods in all engines, update it with one or more engine names and optionally adjust the other parameters.
Kubernetes Results
devin [ ~/repo/dremio-cloud-tools/charts/dremio_v2 ]$ kubectl get pods -n dremio
NAME READY STATUS RESTARTS AGE
dremio-executor-example1-0 0/1 Init:1/2 0 2m35s
dremio-executor-example1-1 1/1 Running 0 2m35s
dremio-executor-example2-0 1/1 Running 0 2m35s
dremio-executor-example2-1 1/1 Running 0 2m35s
dremio-master-0 1/1 Running 0 113s
zk-0 1/1 Running 0 2m35s
zk-1 0/1 ContainerCreating 0 2m35s
zk-2 1/1 Running 0 2m35s
Dremio UI Results
When the Dremio "Node Activity" page is viewed, the Engine name for a node will be shown in the "Engine" column:
Customizing Engines
To change parameters for one specific engine, including Kubernetes resource limits, like CPU, memory and storage, or if you need to change the count of the pods in the engine, find the engineOverride
block in the values.yaml
file.
engineOverride:
example-override1:
cpu: 1
memory: 8000
count: 1
...
volumeSize: 128Gi
...
cloudCache:
enabled: false
Additional Resources
This configuration will allow platform operators to manage rules and queues to route queries and jobs to particular groups of executor nodes. Queues and rules are not detailed in this article, but you can learn more about them in our documentation pages: https://docs.dremio.com/software/advanced-administration/workload-management/