YARN Capacity Scheduler

1. Enable YARN capacity scheduler

yarn-site.xml

<property>
    <name>yarn.resourcemanager.scheduler.class</name>
    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
</property>

2. Setup queues

The CapacityScheduler has a pre-defined queue called root. All queueus in the system are children of the root queue.
The configuration for CapacityScheduler uses a concept called queue path to configure the hierarchy of queues. The queue path is the full path of the queue's hierarchy, starting at root, with . (dot) as the delimiter.

For example, below are 3 level queue path. (Yep, I = NBA fan.)
Starting at "root", "Spurs", "Heat" are the name of level 2 queue;
"Duncan", "Parker" are children of "Spurs", "James","Wade" are children of "Heat".
capacity-scheduler.xml

  <property>
    <name>yarn.scheduler.capacity.root.queues</name>
    <value>Spurs,Heat</value>
    <description>
      Level 2: Children of "root" queue.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.Spurs.queues</name>
    <value>Duncan,Parker</value>
    <description>
       Level 3: Children of "Spurs" queue.
    </description>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.Heat.queues</name>
    <value>James,Wade</value>
    <description>
       Level 3: Children of "Heat" queue.
    </description>
  </property>

3. Resource Allocation

Suppose Spurs beats Heat 60-40 in Finals.
Duncan and Parker each got 50% points of Spurs total points;
James got 90% points of Heat total point while Wade only got 10%.
Please note that:

The sum of capacities for all queues, at each level, must be equal to 100.
Applications in the queue may consume more resources than the queue's capacity if there are free resources, providing elasticity.

capacity-scheduler.xml

  <property>
      <name>yarn.scheduler.capacity.root.Spurs.capacity</name>
      <value>60</value>
      <description>Spurs won with 60% points.</description>
  </property>

  <property>
        <name>yarn.scheduler.capacity.root.Spurs.Duncan.capacity</name>
        <value>50</value>
        <description>Duncan is never old, and got 50% Spurs total points.</description>
  </property>

  <property>
        <name>yarn.scheduler.capacity.root.Spurs.Parker.capacity</name>
        <value>50</value>
        <description>Parker got 50% Spurs total points.</description>
  </property>

  <property>
        <name>yarn.scheduler.capacity.root.Heat.capacity</name>
        <value>40</value>
        <description>Heat only got 40% points.</description>
  </property>

  <property>
        <name>yarn.scheduler.capacity.root.Heat.James.capacity</name>
        <value>90</value>
        <description>James got 90% Heat total points.</description>
  </property>

  <property>
        <name>yarn.scheduler.capacity.root.Heat.Wade.capacity</name>
        <value>10</value>
        <description>Wade only got 10% Heat total points.</description>
  </property>

4. Activate Queues

Imagine in 2024, Duncan and Parker are still fighting together in Spurs, while James and Wade already got retired.
capacity-scheduler.xml

  <property>
    <name>yarn.scheduler.capacity.root.Spurs.state</name>
    <value>RUNNING</value>
    <description>
      The state of the Spurs queue. State can be one of RUNNING or STOPPED.
    </description>
  </property>

  <property>
      <name>yarn.scheduler.capacity.root.Heat.state</name>
      <value>RUNNING</value>
  </property>

  <property>
      <name>yarn.scheduler.capacity.root.Spurs.Duncan.state</name>
      <value>RUNNING</value>
   </property>

  <property>
      <name>yarn.scheduler.capacity.root.Spurs.Parker.state</name>
      <value>RUNNING</value>
  </property>

  <property>
      <name>yarn.scheduler.capacity.root.Heat.James.state</name>
      <value>STOPPED</value>
  </property>

  <property>
      <name>yarn.scheduler.capacity.root.Heat.Wade.state</name>
      <value>STOPPED</value>
  </property>

5. Control Access to Queues

Property acl_submit_applications can be set to "user1,user2 group1,group".
Or "*" (asterisk) to allow access to all users and groups, Or " " (space character) to block access to all users and groups.

  <property>
      <name>yarn.scheduler.capacity.root.acl_submit_applications</name>
      <value>hadoop,yarn,mapred,hdfs</value>
      <description>
             The ACL of who can submit jobs to the root queue.
      </description>
   </property>

  <property>
    <name>yarn.scheduler.capacity.root.Spurs.acl_submit_applications</name>
    <value>tim,tony</value>
  </property>

  <property>
    <name>yarn.scheduler.capacity.root.Heat.acl_administer_queue</name>
    <value>lebron,dwyane</value>
  </property>

  <property>
      <name>yarn.scheduler.capacity.root.Spurs.Duncan.acl_submit_applications</name>
      <value>tim</value>
  </property>

  <property>
        <name>yarn.scheduler.capacity.root.Spurs.Parker.acl_submit_applications</name>
        <value>tony</value>
  </property>

  <property>
         <name>yarn.scheduler.capacity.root.Heat.James.acl_submit_applications</name>
         <value>lebron</value>
  </property>
  
  <property>
         <name>yarn.scheduler.capacity.root.Heat.Wade.acl_submit_applications</name>
         <value>dwyane</value>
  </property>

6. User Resource Limit of Queue

yarn.scheduler.capacity.<queue path>.minimum-user-limit-percent
Suppose it is set to 25.

1 user	This user can get 100% queue capacity.
2 users	Each user can get 50% queue capacity at most.
3 users	Each user can get 33% queue capacity at most.
4 users	Each user can get 25% queue capacicy at most.
5 users	5th user must wait for resource to be freed.

7. User Resource Hard Limit.

yarn.scheduler.capacity.<queue path>.user-limit-factor
Different values means:

1	Any user can only occupy at most the queue's configured capacity.
2	Any user can only occupy at most twice of queue's configured capacity.
0.5	Any user can only occupy at most half of queue's configured capacity.

This is because, Applications in the queue may consume more resources than the queue's capacity if there are free resources, providing elasticity.
This can avoid one queue growing to a monster.

8. Refresh Queues

yarn rmadmin -refreshQueues

9. Show queues

[root@hdm]# mapred queue -list
======================
Queue Name : Heat
Queue State : running
Scheduling Info : Capacity: 40.0, MaximumCapacity: 1.0, CurrentCapacity: 0.0
    ======================
    Queue Name : James
    Queue State : stopped
    Scheduling Info : Capacity: 90.0, MaximumCapacity: 1.0, CurrentCapacity: 0.0
    ======================
    Queue Name : Wade
    Queue State : stopped
    Scheduling Info : Capacity: 10.0, MaximumCapacity: 1.0, CurrentCapacity: 0.0
======================
Queue Name : Spurs
Queue State : running
Scheduling Info : Capacity: 60.000004, MaximumCapacity: 1.0, CurrentCapacity: 0.0
    ======================
    Queue Name : Duncan
    Queue State : running
    Scheduling Info : Capacity: 50.0, MaximumCapacity: 1.0, CurrentCapacity: 0.0
    ======================
    Queue Name : Parker
    Queue State : stopped
    Scheduling Info : Capacity: 50.0, MaximumCapacity: 1.0, CurrentCapacity: 0.0

10. Show user ACLs

[tim@hdm ~]$ mapred queue -showacls
Queue acls for user :  tim

Queue  Operations
=====================
root  ADMINISTER_QUEUE
Heat  ADMINISTER_QUEUE
James  ADMINISTER_QUEUE
Wade  ADMINISTER_QUEUE
Spurs  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
Duncan  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS
Parker  ADMINISTER_QUEUE,SUBMIT_APPLICATIONS

If Kerberos is enabled, below steps need to be done firstly:
Firstly go to KDC host to create the principal for user "tim":

[root@admin ~]# kadmin.local
Authenticating as principal root/admin@OPENKBINFO.COM with password.
kadmin.local:  addprinc tim
WARNING: no policy specified for tim@OPENKBINFO.COM; defaulting to no policy
Enter password for principal "tim@OPENKBINFO.COM":
Re-enter password for principal "tim@OPENKBINFO.COM":
Principal "tim@OPENKBINFO.COM" created.

Then go to "hdm" logon "tim" user:

[root@hdm ]# su - tim

[tim@hdm ~]# kinit
Password for tim@OPENKBINFO.COM:

11. Test Yarn Job

Using "-D mapreduce.job.queuename=<Queue name>" to submit the job to the queue.

hadoop fs -rmr /tmp/test_input
hadoop fs -mkdir /tmp/test_input
hadoop fs -rmr /user/tim/test_output
hadoop fs -copyFromLocal /usr/lib/gphd/hadoop/CHANGES.txt /tmp/test_input
yarn jar /usr/lib/gphd/hadoop-mapreduce/hadoop-mapreduce-examples.jar wordcount -D mapreduce.job.queuename=Duncan /tmp/test_input /user/tim/test_output
hadoop fs -cat /user/tim/test_output/part*

If you try to submit to wrong queues, below error will show up:

ERROR security.UserGroupInformation: PriviledgedActionException as:tim@OPENKBINFO.COM (auth:KERBEROS) cause:java.io.IOException: Failed to run job : org.apache.hadoop.security.AccessControlException: User tim cannot submit applications to queue root.Heat.James
java.io.IOException: Failed to run job : org.apache.hadoop.security.AccessControlException: User tim cannot submit applications to queue root.Heat.James

Sunday, June 8, 2014