Stop accepting new jobs in a hadoop cluster (ACL)

To stop accepting new MR jobs in a hadoop cluster you have to enable ACL's first. If you've done that, you can specify a single character queue ACL (' ' = a space!). Since mapred-queue-acls.xml is polled regularly you can dynamically change the queue in a running system . Useful for ops related work (setting into maintenance, extending / decommission nodes and such things).

Enable ACL's

Edit the config file ($HADOOP/conf/mapred-queue-acls.xml) to fit your needs:

<configuration>
 <property>
   <name>mapred.queue.default.acl-submit-job</name>
   <value>user1,user2,group1,group2,admins</value>
 </property>

 <property>
   <name>mapred.queue.default.acl-administer-jobs</name>
   <value>admins</value>
 </property>

</configuration>


Enable an ACL driven cluster by editing the value of mapred.acls.enabled in conf/mapred-site.xml and setting to true.

Now edit simply the value of mapred.queue.default.acl-submit-job and replace user1,user2,group1,group2 with ' ':

<configuration>
 <property>
   <name>mapred.queue.default.acl-submit-job</name>
   <value> </value>
 </property>

 <property>
   <name>mapred.queue.default.acl-administer-jobs</name>
   <value>admins</value>
 </property>

</configuration>


This stops all users to submit new jobs, but lets the started jobs running.

Comments

Popular posts from this blog

Export HDFS over CIFS (Samba3)

Hive query shows ERROR "too many counters"

Connect to HiveServer2 with a kerberized JDBC client (Squirrel)