Overview
You may need to investigate a Zookeeper issue (crash, slowness, process hanging, etc...) and you will find that in some instances (eg. AWSE environment), Zookeeper is running as a service, with events logged in the journal and/or messages/syslog.
You can enable debug logging for the Zookeeper process and push it within its own log and location.
Applies To
All versions pre Dremio 22.0.0 (Zookeeper was upgraded to version 3.8 then)
Details
To do so, you will need to add the following 2 lines at the end of /opt/zookeeper/bin/zkEnv.sh :
ZOO_LOG_DIR="/var/log/zookeeper"
ZOO_LOG4J_PROP="DEBUG,ROLLINGFILE"
You may need to create a new var/log/zookeeper directory with RW permissions before restarting the zookeeper service with command sudo systemctl restart zookeeper.service
Keep in mind log rotation, dealt with inside your /opt/zookeeper/conf/log4j.properties file via:
log4j.appender.ROLLINGFILE.MaxFileSize=10MBwhere you can choose to increase or reduce the size according to your needs.
Further Reading
These steps will not work from Dremio 22.0.0 onward (where Zookeeper was upgraded to version 3.8) and you will need to add the following in your /opt/zookeeper/conf/logback.xml where you can change the following:
<property name="zookeeper.console.threshold" value="DEBUG" />
<property name="zookeeper.log.dir" value="/var/log/zookeeper" />
<property name="zookeeper.log.file" value="zookeeper.log" />
<property name="zookeeper.log.threshold" value="DEBUG" />
<property name="zookeeper.log.maxfilesize" value="256MB" />
<property name="zookeeper.log.maxbackupindex" value="20" />
<root level="DEBUG">
<appender-ref ref="ROLLINGFILE" />
</root>