Saturday, June 14, 2014

HDFS YARN cheat sheet

HDFS

1. HDFS report
hdfs dfsadmin -report
2. Namenode HA
hdfs haadmin -failover nn2 nn1
hdfs haadmin -getServiceState nn1
hdfs haadmin -getServiceState nn2
3. Safe mode
hdfs dfsadmin -safemode get
hdfs dfsadmin -safemode enter
hdfs dfsadmin -safemode leave
4. fsck
hdfs fsck /
hadoop fsck / -move
hadoop fsck / -delete
hadoop fsck / -files -blocks -locations
5. Namenode recover
hdfs namenode -recover
6. Secondary namenode
hdfs secondarynamenode -checkpoint force
7. Offline fsimage/edits viewer
hdfs oev -i edits_inprogress_0000000000000011129 -o /tmp/edits.xml
hdfs oiv -i fsimage_0000000000000011128 -o /tmp/fsimage.out
8. Replication factor
hdfs dfs -setrep 3 filename
9. hdfs dfs -stat <file path>
%b  Size of file in bytes
%F  Will return "file", "directory", or "symlink" depending on the type of inode
%g  Group name
%n  Filename
%o  HDFS Block size in bytes ( 128MB by default )
%r  Replication factor
%u   Username of owner
%y  Formatted mtime of inode
%Y  UNIX Epoch mtime of inode
10. Configuring JAVA
alternatives --install /usr/bin/java java /usr/java/default/bin/java 1
11. RWtest
hadoop jar ./lib/maprfs-core-4.0.1-mapr-tests.jar com.mapr.fs.RWSpeedTest /hugefile/3 26000000 maprfs:///
12. checknative
hadoop checknative -a

==

YARN

1. Yarn node list
yarn node -list
2. Yarn Queues
yarn rmadmin -refreshQueues
3. Jobs List
mapred job -list
4. Troubleshoot
jstack -J-d64 -m <pid>
5.Print effective parameters
hadoop conf-details print-all-effective-properties

MRV1

1. Set debug logging for jobtracker
maprcli setloglevel jobtracker -classname org.apache.hadoop.mapred.JobTracker -loglevel DEBUG -node n2a  -port 50030
maprcli setloglevel jobtracker -classname org.apache.hadoop.mapred.JobTracker -loglevel INFO -node n2a  -port 50030

hadoop daemonlog -setlevel n2a:50030 org.apache.hadoop.mapred.JobTracker DEBUG    
hadoop daemonlog -setlevel n2a:50030 org.apache.hadoop.mapred.JobTracker INFO

==

No comments:

Post a Comment

Popular Posts