How to read or dump the Zookeeper Transaction Log and Snapshots

Goal:

How to read or dump the Zookeeper Transaction Log and Snapshots.

Env:

Zookeeper 3.4.5 on MapR 5.1

Solution:

When troubleshooting, we may want to dig into the Zookeeper Transaction Log and Snapshots which are in binary.
If we want to get most readable information out of them, here are the commands:

1. Zookeeper Transaction Log

java -cp /opt/mapr/lib/zookeeper-3.4.5-mapr-1503.jar:/opt/mapr/lib/log4j-1.2.17.jar:/opt/mapr/lib/slf4j-log4j12-1.7.12.jar:/opt/mapr/lib/slf4j-api-1.7.12.jar org.apache.zookeeper.server.LogFormatter log.100000001

The location of the Zookeeper transaction log by default on mapr env is:
/opt/mapr/zkdata/version-2
The output will print how many transactions are recorded in this Zookeeper Transaction Log in the end, for example:

EOF reached after 1404 txns.

To double confirm that, you can also "grep session" from the output, and count it by yourself:

#  java -cp /opt/mapr/lib/zookeeper-3.4.5-mapr-1503.jar:/opt/mapr/lib/log4j-1.2.17.jar:/opt/mapr/lib/slf4j-log4j12-1.7.12.jar:/opt/mapr/lib/slf4j-api-1.7.12.jar org.apache.zookeeper.server.LogFormatter log.100000001  |grep session|wc -l
1404

Or if you want to know which Hadoop Ecosystem has the most calls to Zookeeper, one way is to check the "SetData" calls. For example:

#  java -cp /opt/mapr/lib/zookeeper-3.4.5-mapr-1503.jar:/opt/mapr/lib/log4j-1.2.17.jar:/opt/mapr/lib/slf4j-log4j12-1.7.12.jar:/opt/mapr/lib/slf4j-api-1.7.12.jar org.apache.zookeeper.server.LogFormatter log.100000001|grep session|grep setData|awk '{print $12}'|awk -F/ '{print $3}'|sort|uniq -c|sort -n
     15 historyserver
     19 spark-historyserver
     24 resourcemanager
     26 controlnodes
     26 webserver
     39 hoststats
     39 kvstore
     56 cldb
     57 nodemanager
     84 nfs

 2. Snapshots

java -cp /opt/mapr/lib/zookeeper-3.4.5-mapr-1503.jar:/opt/mapr/lib/log4j-1.2.17.jar:/opt/mapr/lib/slf4j-log4j12-1.7.12.jar:/opt/mapr/lib/slf4j-api-1.7.12.jar org.apache.zookeeper.server.SnapshotFormatter snapshot.0

The output will print the znodes list at that time, you can just "grep '/'" to get them, for example:

# java -cp /opt/mapr/lib/zookeeper-3.4.5-mapr-1503.jar:/opt/mapr/lib/log4j-1.2.17.jar:/opt/mapr/lib/slf4j-log4j12-1.7.12.jar:/opt/mapr/lib/slf4j-api-1.7.12.jar org.apache.zookeeper.server.SnapshotFormatter snapshot.10000057c|grep '/'
/
/historyserver_locks
/nodes
/nodes/v5.poc.com
/nodes/v5.poc.com/services
...
...