Summary
This article will guide you through how to enable debug log level for the Zookeeper process
Reported Issue
You may need to investigate a Zookeeper issue (crash, slowness, process hanging, etc...) and you will find that in some instances (eg. AWSE environment), Zookeeper is running as a service, with events logged in the journal and/or messages/syslog.
Overview
You can enable debug logging for the Zookeeper process and push it within its own log and location.
Relevant Versions Tools and Integrations
All versions pre Dremio 22.0.0 (Zookeeper was upgraded to version 3.8 then)
Steps to Resolve
To do so, you will need to add the following 2 lines at the end of /opt/zookeeper/bin/zkEnv.sh
:
ZOO_LOG_DIR="/var/log/zookeeper"
ZOO_LOG4J_PROP="DEBUG,ROLLINGFILE"
You may need to create a new var/log/zookeeper
directory with RW permissions before restarting the zookeeper
service with command sudo systemctl restart zookeeper.service
Common Challenges
Keep in mind log rotation, dealt with inside your /opt/zookeeper/conf/log4j.properties
file via:
log4j.appender.ROLLINGFILE.MaxFileSize=10MB
where you can choose to increase or reduce the size according to your needs.
Additional Resources
These steps will not work from Dremio 22.0.0 onward (where Zookeeper was upgraded to version 3.8) and you will need to add the following in your /opt/zookeeper/conf/logback.xml where you can change the following:
<property name="zookeeper.console.threshold" value="DEBUG" />
<property name="zookeeper.log.dir" value="/var/log/zookeeper" />
<property name="zookeeper.log.file" value="zookeeper.log" />
<property name="zookeeper.log.threshold" value="DEBUG" />
<property name="zookeeper.log.maxfilesize" value="256MB" />
<property name="zookeeper.log.maxbackupindex" value="20" />
<root level="DEBUG">
<appender-ref ref="ROLLINGFILE" />
</root>