Tuesday, June 24, 2014

Namenode HA configurations

Below are the configurations for Namenode HA.
The env is CDH5 managed by CM5.

hdfs-site.xml

  <property>
    <name>dfs.nameservices</name>
    <value>nameservice1</value>
  </property>

  <property>
    <name>dfs.ha.namenodes.nameservice1</name>
    <value>namenode129,namenode139</value>
  </property>

  <property>
    <name>dfs.namenode.name.dir.nameservice1.namenode129</name>
    <value>file:///data/1/dfs/nn,file:///data/2/dfs/nn</value>
  </property>
  <property>
    <name>dfs.namenode.name.dir.nameservice1.namenode139</name>
    <value>file:///data/1/dfs/nn,file:///data/2/dfs/nn</value>
  </property>
  
  <property>
    <name>dfs.namenode.rpc-address.nameservice1.namenode129</name>
    <value>admin.xxx.com:8020</value>
  </property>
  <property>
    <name>dfs.namenode.servicerpc-address.nameservice1.namenode129</name>
    <value>admin.xxx.com:8022</value>
  </property>
  <property>
    <name>dfs.namenode.http-address.nameservice1.namenode129</name>
    <value>admin.xxx.com:50070</value>
  </property>
  <property>
    <name>dfs.namenode.https-address.nameservice1.namenode129</name>
    <value>admin.xxx.com:50470</value>
  </property>

  <property>
    <name>dfs.namenode.rpc-address.nameservice1.namenode139</name>
    <value>hdw3.xxx.com:8020</value>
  </property>
  <property>
    <name>dfs.namenode.servicerpc-address.nameservice1.namenode139</name>
    <value>hdw3.xxx.com:8022</value>
  </property>
  <property>
    <name>dfs.namenode.http-address.nameservice1.namenode139</name>
    <value>hdw3.xxx.com:50070</value>
  </property>
  <property>
    <name>dfs.namenode.https-address.nameservice1.namenode139</name>
    <value>hdw3.xxx.com:50470</value>
  </property>

  <property>
    <name>dfs.namenode.shared.edits.dir.nameservice1.namenode129</name>
    <value>qjournal://admin.xxx.com:8485;hdm.xxx.com:8485;hdw1.xxx.com:8485;hdw2.xxx.com:8485;hdw3.xxx.com:8485/nameservice1</value>
  </property>
  <property>
    <name>dfs.namenode.shared.edits.dir.nameservice1.namenode139</name>
    <value>qjournal://admin.xxx.com:8485;hdm.xxx.com:8485;hdw1.xxx.com:8485;hdw2.xxx.com:8485;hdw3.xxx.com:8485/nameservice1</value>
  </property>

  <property>
    <name>dfs.client.failover.proxy.provider.nameservice1</name>
    <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
  </property>

  <property>
    <name>dfs.ha.fencing.methods</name>
    <value>shell(true)</value>
  </property>

  <property>
    <name>dfs.ha.automatic-failover.enabled.nameservice1</name>
    <value>true</value>
  </property>

<!-- Journal Node specific-->
  <property>
    <name>dfs.journalnode.edits.dir</name>
    <value>/data/edits</value>
  </property>

<!-- Cloudera Manager specific-->
  <property>
    <name>dfs.ha.fencing.cloudera_manager_agent.url.namenode139</name>
    <value>http://hdw3.xxx.com:9000/fence/fence?name=hdfs-NAMENODE&amp;host=hdw3.xxx.com&amp;port=8020</value>
  </property>
  <property>
    <name>dfs.ha.fencing.cloudera_manager_agent.url.namenode129</name>
    <value>http://admin.xxx.com:9000/fence/fence?name=hdfs-NAMENODE&amp;host=admin.xxx.com&amp;port=8020</value>
  </property>

core-site.xml

  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://nameservice1</value>
  </property>

hbase-site.xml

  <property>
    <name>hbase.rootdir</name>
    <value>hdfs://nameservice1/hbase</value>
  </property>

hue.ini

fs_defaultfs=hdfs://nameservice1

Hive metastore:

/opt/cloudera/parcels/CDH/lib/hive/bin/metatool -listFSRoot

Listing FS Roots..
hdfs://nameservice1/user/hive/warehouse

3 comments:

  1. A clear overview of Namenode HA configurations. The practical points on failover, reliability, and avoiding downtime make this a useful read for Hadoop users. baked lasagna

    ReplyDelete
  2. Great explanation of Namenode HA configurations! The clear points on high availability, failover, and reliability make this topic much easier to understand and apply in real-world Hadoop environments. family business tax planning

    ReplyDelete

Popular Posts