Summary
This article explains how to change the memory settings (including direct and heap memory) on individual AWSE executor engines based on the AWSE instance type. This is done by modifying the executor configuration files on the coordinator node and then pushing out the changes to all the executor nodes.
Reported Issue
Currently the Dremio AWSE administrator is limited to using the same executor configuration via the executor_customization.sh script for all executors within all engines in their Dremio AWSE cluster. However due to varying workloads, they may need bigger instance types for those WLM queues setup for resource/memory intensive query types whilst having smaller executor engines for queries that do not require significant compute resources.
Overview
The AWSE executor customization script offers the ability to change how executor nodes are configured by modifying the executor configuration files on the coordinator node and then pushing out the changes to all the executor nodes per the following documentation page: https://docs.dremio.com/25.x/get-started/cluster-deployments/deployment-models/amazon-deployments/aws/configure/aws-edition-executors. However this is only a guide and so the below aids in further configuring and tuning this to the above desired requirements.
Relevant Versions Tools and Integrations
All versions of Dremio up to and including v25.
Steps to Resolve
The below example executor_customization.sh script checks the executor instance type and if it is m5d.8xlarge, then the script adjusts the executor heap to 16 GB (i.e copies a 16GB heap memory setting dremio-env file to the executor).
The script also contains a check to see if the instance type is anything else other than m5d.8xlarge, and if so the heap is not adjusted which provides the flexibility of engine specific memory settings.
As this is only an example, the script can be adjusted to suit any other desired AWS instance types.
#!/bin/bash
#
# For running as part of the executor_customization.sh script
# Detecting the aws instance type of executor
# Based on instance type, adjusting heap memory (xmx) as necessary by copying through appropriate dremio-env file to executor
instanceType=$(ec2-metadata --instance-type | sed 's/instance-type: //g')
if [ $instanceType = "m5d.8xlarge" ];
then
echo "Detected instance type as m5d.8xlarge. Will adjust executor heap to 16gb"
cp -rf /var/dremio_efs/executor/etc/dremio/heap16gb-dremio-env /etc/dremio/dremio-env
elif [ $instanceType = "m5d.2xlarge" ] ; then
echo "Will not adjust executor heap."
fi;
Example output dremio-env configuration file:
heap16gb-dremio-env:
DREMIO_LOG_DIR=/var/log/dremio DREMIO_PID_DIR=/var/run/dremio DREMIO_GC_LOGS_ENABLED="yes" DREMIO_GC_OPTS="-XX:+UseG1GC" DREMIO_JAVA_VERSION_CHECK="true" #DREMIO_JAVA_EXTRA_OPTS= DREMIO_EXTRA_CLASSPATH=/var/dremio_efs/thirdparty/* DREMIO_MAX_MEMORY_SIZE_MB=104448 # Let netty manage direct memory, this avoids cleaners triggering GCs DREMIO_NETTY_MAX_DIRECT_MEMORY="-1" DREMIO_JAVA_SERVER_EXTRA_OPTS="-Xms16g -Xmx16g"
Next Steps
The executor_customization.sh script should be placed in the following coordinator directory: /var/dremio_efs/executor_customization.sh and should be given 755 permissions.
The heap16gb-dremio-env file should be placed in the following coordinator directory /var/dremio_efs/executor/etc/dremio/heap16gb-dremio-env.
Additional Resources
Configuring AWSE Executor Nodes: https://docs.dremio.com/25.x/get-started/cluster-deployments/deployment-models/amazon-deployments/aws/configure/aws-edition-executors/