Friday, December 14, 2012

Impala and Kerberos

First, Impala is beta software and has some limitations. Stay tuned and test this, you'll see it can be change your BI world dramatically.

What is Impala? 
Impala provides fast, interactive SQL queries directly on your Apache Hadoop data stored in HDFS or HBase. In addition to using the same unified storage platform, Impala also uses the same metadata, SQL syntax (Hive SQL), ODBC driver and user interface (Hue Beeswax) as Apache Hive. This provides a familiar and unified platform for batch-oriented or real-time queries.
(https://ccp.cloudera.com/display/IMPALA10BETADOC/Introducing+Cloudera+Impala)
You can build Impala by source (https://github.com/cloudera/impala) or you can grab them by using yum on a RHEL / CentOS 6x server. Imapla doesn't support RHEL / CentOS prior 6, since the most part of Impala is written in C++.

I choose the rpm-version for this article, but the compiled version will work in the same manner. To grab impala directly per yum setup a new repository:

#> cat /etc/yum.repos.d/impala.repo
[cloudera-impala]
name=Impala
baseurl=http://beta.cloudera.com/impala/redhat/6/x86_64/impala/0/
gpgkey = http://beta.cloudera.com/impala/redhat/6/x86_64/impala/RPM-GPG-KEY-cloudera
gpgcheck = 1

and install impala and all needed libs per yum:

yum install impala impala-shell cyrus-sasl-devel cyrus-sasl-gssapi gcc-c++ gcc c++ python-setuptools -y && easy_install sasl

You should use the newest JDK from Oracle and you have to install it along your cluster, in this article jdk-6u37-linux-x64-rpm.bin was the actual release. Note, you have to install the JDK after you have installed Impala per yum, as the dependencies install OpenJDK too. To avoid the using of OpenJDK point your system(s) per alternative to the release you want to use:

alternatives --install /usr/bin/javaws javaws /usr/java/latest/jre/bin/javaws 20000
alternatives --install /usr/bin/java java /usr/java/latest/jre/bin/java 20000


To be sure you're running with the JDK you've installed ago you should check it:

java -version
java version "1.6.0_37"
Java(TM) SE Runtime Environment (build 1.6.0_37-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01, mixed mode)


One of the things which can be go wrong are some missed libs, they are dynamically linked from impalad and not present in the default library stores. Check it with ldd and link the missed libs into /usr/lib64/, in my case I did:

ln -s /usr/java/jdk1.6.0_37/jre/lib/amd64/server/libjvm.so /usr/lib64/libjvm.so
ln -s /usr/java/jdk1.6.0_37/jre/lib/amd64/libjsig.so /usr/lib64/libjsig.so
ln -s /usr/lib/impala/lib/libhdfs.so.0.0.0 /usr/lib64/libhdfs.so.0.0.0


you should check if you find any missed libraries by using ldd (ldd /usr/lib/impala/sbin/impalad) .

Copy your hive, hdfs and hbase config files into the config directory of Impala and create a log4.properties file within $IMPALA_HOME/conf/:

log.threshold=INFO
main.logger=FA
impala.root.logger=${log.threshold},${main.logger}
log4j.rootLogger=${impala.root.logger}
log.dir=/var/log/impalad
log.file=impalad.INFO
log4j.appender.FA=org.apache.log4j.FileAppender
log4j.appender.FA.File=${log.dir}/${log.file}
log4j.appender.FA.layout=org.apache.log4j.PatternLayout
log4j.appender.FA.layout.ConversionPattern=%p%d{MMdd HH:mm:ss.SSS'000'} %t %c] %m%n
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n


The config directory within impalas home should have the following files present, to determine which home-directory the user impala use check it with "echo ~impala".

ll /usr/lib/impala/conf/
total 24
-rw-r--r-- 1 root root 1243 Dec 10 14:59 core-site.xml
-rw-r--r-- 1 root root 4596 Sep 2 09:35 hdfs-site.xml
-rw-r--r-- 1 root root 1157 Dec 10 10:36 hive-site.xml
-rw------- 1 impala impala 594 Dec 11 12:29 impala.keytab
-rw-r--r-- 1 root root 647 Dec 11 12:31 log4j.properties


Sync the content of the directory to all other nodes in your cluster.

Kerberos integration
If you use RHEL kerberos KDC packages you have to tweak your principals. From RHEL4 on principals getting a default renew_lifetime by zero. Means, you can get a renewable ticket, but you can't renew this.

To solve this you have to modify the krbtgt principal as well all other principals who should have the availability to renew their tickets.
kadmin.local: modprinc -maxrenewlife 1day krbtgt/ALO.ALT@ALO.ALT
kadmin.local: addprinc -randkey -maxrenewlife 1day +allow_renewable impala/hadoop1.alo.alt@ALO.ALT

Export the keytab (xst -norandkey -k impala.keytab impala/hadoop1.alo.alt@ALO.ALT HTTP/hadoop1.alo.alt@ALO.ALT), place it in $IMPALA_HOME/conf and obtain a renewable ticket with
sudo -u impala kinit -r 1day -k -t /usr/lib/impala/conf/impala.keytab impala/hadoop1.alo.alt@ALO.ALT

I created a poor startscript to check if all is working as expected and start statestore as well as  impalad on your server:

CONF=/usr/lib/impala/conf
USER=impala
PWD=`echo ~$USER`
HOST=`hostname`
REALM=ALO.ALT
export GLOG_minloglevel=0
export GLOG_logbuflevel=-1
export GLOG_log_dir=/var/log/impala
export GLOG_max_log_size=200

mkdir -p /var/log/impala
chown -R impala: /var/log/impala
# obtain a new ticket
sudo -u impala kinit -r 1day -k -t $CONF/$USER.keytab $USER/$HOST@$REALM
#start it up
statestored -state_store_port=24000 -enable_webserver=true -webserver_port=25010 -log_filename=impala-state-store -principal=$USER/$HOST@$REALM -keytab_file=$CONF/impala.keytab &

impalad -state_store_host=hadoop1.alo.alt -nn=hadoop1.alo.alt -nn_port=9000 -hostname=hadoop1.alo.alt -ipaddress=192.168.56.101 -enable_webserver=true -webserver_port=25000 -principal=$USER/$HOST@$REALM -keytab_file=$CONF/impala.keytab -kerberos_ticket_life=36000 -log_filename=impala &


To control if all is running well you can now point your browser to the configured webservices:
statestore: http://<statestore-server>:25010
impalad: http://<impala-server>:25000

Both services deliver a bunch of monitoring features, as example you can grab metrics from the /metrics endpoint.

Using impala-shell with kerberos
To use impala shell with kerberos you have to get a valid ticket for your user first and have to invoke the shell per impala-shell -k. Note, that on all clients you have to install python sasl (best way per easy_install sasl)

[~]$ impala-shell -k
Using service name 'impala' for kerberos
Welcome to the Impala shell. Press TAB twice to see a list of available commands.

Copyright (c) 2012 Cloudera, Inc. All rights reserved.

(Build version: Impala v0.3 (3cb725b) built on Fri Nov 23 13:51:59 PST 2012)
[Not connected] > connect hadoop1:21000
[hadoop1:21000] > show tables
hbase_test
hbase_test2
hivetest1
hivetest2
[hadoop1:21000] >