Tuesday, May 15, 2012

Stop accepting new jobs in a hadoop cluster (ACL)

To stop accepting new MR jobs in a hadoop cluster you have to enable ACL's first. If you've done that, you can specify a single character queue ACL (' ' = a space!). Since mapred-queue-acls.xml is polled regularly you can dynamically change the queue in a running system . Useful for ops related work (setting into maintenance, extending / decommission nodes and such things).

Enable ACL's

Edit the config file ($HADOOP/conf/mapred-queue-acls.xml) to fit your needs:

<configuration>
 <property>
   <name>mapred.queue.default.acl-submit-job</name>
   <value>user1,user2,group1,group2,admins</value>
 </property>

 <property>
   <name>mapred.queue.default.acl-administer-jobs</name>
   <value>admins</value>
 </property>

</configuration>


Enable an ACL driven cluster by editing the value of mapred.acls.enabled in conf/mapred-site.xml and setting to true.

Now edit simply the value of mapred.queue.default.acl-submit-job and replace user1,user2,group1,group2 with ' ':

<configuration>
 <property>
   <name>mapred.queue.default.acl-submit-job</name>
   <value> </value>
 </property>

 <property>
   <name>mapred.queue.default.acl-administer-jobs</name>
   <value>admins</value>
 </property>

</configuration>


This stops all users to submit new jobs, but lets the started jobs running.