Dienstag, 14. April 2015

Hive on Spark at CDH 5.3

However, since Hive on Spark is not (yet) officially supported by Cloudera some manual steps are required to get Hive on Spark within CDH 5.3 working. Please note that there are four important requirements additionally to the hands-on work:
  1. Spark Gateway nodes needs to be a Hive Gateway node as well
  2. In case the client configurations are redeployed, you need to copy the hive-site.xml again
  3. In case CDH is upgraded (also for minor patches, often updated without noticing you), you need to adjust the class paths
  4. Hive libraries need to be present on all executors (CM should take care of this automatically)
Login to your spark server(s) and copy the running hive-site.xml to spark:

cp /etc/hive/conf/hive-site.xml /etc/spark/conf/

Start your spark shell with (replace <CDH_VERSION> with your parcel version, e.g. 5.3.2-1.cdh5.3.2.p0.10) and load the hive context within spark-shell:

spark-shell --master yarn-client --driver-class-path "/opt/cloudera/parcels/CDH-<CDH_VERSION>/lib/hive/lib/*" --conf spark.executor.extraClassPath="/opt/cloudera/parcels/CDH-<CDH_VERSION>/lib/hive/lib/*"
..
scala> val hive = new org.apache.spark.sql.hive.HiveContext(sc)
sql: org.apache.spark.sql.hive.HiveContext = org.apache.spark.sql.hive.HiveContext@1c966488

scala> var s1 = hive.sql("SELECT COUNT(*) FROM sample_07").collect()
s1: Array[org.apache.spark.sql.Row] = Array([823])

Montag, 16. Februar 2015

Hadoop and trusted MiTv5 Kerberos with Active Directory

For actuality here a example how to enable an MiTv5 Kerberos <=> Active Directory trust just from scratch. Should work out of the box, just replace the realms:

HADOOP1.INTERNAL = local server (KDC)
ALO.LOCAL = local kerberos realm
AD.REMOTE = AD realm

with your servers. The KDC should be inside your hadoop network, the remote AD can be somewhere.

1. Install the bits

At the KDC server (CentOS, RHEL - other OS' should have nearly the same bits):
yum install krb5-server krb5-libs krb5-workstation -y

At the clients (hadoop nodes):
yum install krb5-libs krb5-workstation -y

Install Java's JCE policy (see Oracle documentation) on all hadoop nodes.

2. Configure your local KDC

/etc/krb5.conf

[libdefaults]
default_realm = ALO.LOCAL
dns_lookup_realm = false
dns_lookup_kdc = false
kdc_timesync = 1
ccache_type = 4
forwardable = true
proxiable = true
fcc-mit-ticketflags = true
max_life = 1d
max_renewable_life = 7d
renew_lifetime = 7d
default_tgs_enctypes = aes128-cts arcfour-hmac
default_tkt_enctypes = aes128-cts arcfour-hmac

[realms]
ALO.LOCAL = {
kdc = hadoop1.
internal:88
admin_server = hadoop1.internal:749
max_life = 1d
max_renewable_life = 7d
}
AD.REMOTE = {
kdc = ad.remote.internal:88
admin_server = ad.remote.internal:749
max_life = 1d
max_renewable_life = 7d
}

[domain_realm]
alo.local = ALO.LOCAL
.alo.local = ALO.LOCAL

ad.internal = AD.INTERNAL
.ad.internal = AD.INTERNAL

[logging]
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmin.log
default = FILE:/var/log/krb5lib.log


/var/kerberos/krb5kdc/kdc.conf

[kdcdefaults]
kdc_ports = 88
kdc_tcp_ports = 88

[realms]
ALO.LOCAL = {
#master_key_type = aes256-cts
acl_file = /var/kerberos/krb5kdc/kadm5.acl
dict_file = /usr/share/dict/words
admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
}
 
/var/kerberos/krb5kdc/kadm5.acl
*/admin@ALO.ALT *

Create the realm on your local KDC and start the services
kdb5_util create -s -r ALO.LOCAL
service kadmin restart
service krb5kdc restart
chkconfig kadmin on
chkconfig krb5kdc on

Create the admin principal
kadmin.local -q "addprinc root/admin"

3. Create the MiTv5 trust in AD

Using the Windows - Power(!sic) - Shell
ksetup /addkdc ALO.LOCAL HADOOP1.INTERNAL
netdom trust ALO.LOCAL /DOMAIN: AD.REMOTE /add /realm /passwordt: passw0rd
ksetup /SetEncTypeAttr ALO.LOCAL RC4-HMAC-MD5 AES128-CTS-HMAC-SHA1-96 AES256-CTS-HMAC-SHA1-96 DES-CBC-CRC DES-CBC-MD5

=> On Windows 2003 this works, too:
ktpass /ALO.LOCAL /DOMAIN:AD.REMOTE /TrustEncryp aes128-cts arcfour-hmac

=> On Windows 2008 you have to add:
ksetup /SetEncTypeAttr ALO.LOCAL aes128-cts arcfour-hmac

4. Create the AD trust in MiTv5
kadmin.local: addprinc krbtgt/ALO.LOCAL@AD.REMOTE
password: passw0rd

5. Configure hadoop's mapping rules

core-site.xml

<property>
<name>hadoop.security.auth_to_local</name>
<value>RULE:[1:$1@$0](.*@\QAD.REMOTE\E$)s/@\QAD.REMOTE\E$//
RULE:[2:$1@$0](.*@\QAD.REMOTE\E$)s/@\QAD.REMOTE\E$//
DEFAULT</value>
</property>

Done. Now you should be able to get an ticket from your AD which let you work with your hadoop installation:

#> kinit alo.alt@AD.REMOTE
password:
#> klist
Ticket cache: FILE:/tmp/krb5cc_500
Default principal: alo.alt@AD.REMOTE

Montag, 9. Februar 2015

Hadoop based SQL engines

Apache Hadoop comes more and more into the focus of business critical architectures and applications. Naturally SQL based solutions are the first to get considered, but the market is evolving and new tools are coming up, but leaving unnoticed.

Listed below an overview over currently available Hadoop based SQL technologies. The must haves are:
Open Source (various contributors), low-latency querying possible, supporting CRUD (mostly!) and statements like CREATE, INSERT INTO, SELECT * FROM (limit..), UPDATE Table SET A1=2 WHERE, DELETE, and DROP TABLE.

Apache Hive (SQL-like, with interactive SQL (Stinger)
Apache Drill (ANSI SQL support)
Apache Spark (Spark SQL, queries only, add data via Hive, RDD or Parquet)
Apache Phoenix (built atop Apache HBase, lacks full transaction support, relational operators and some built-in functions)
Presto from Facebook (can query Hive, Cassandra, relational DBs & etc. Doesn't seem to be designed for low-latency responses across small clusters, or support UPDATE operations. It is optimized for data warehousing or analytics¹)
VoltDB (ACID compatible, ANSI SQL near 92, low-latency multi query engine)
SQL-Hadoop via MapR community edition (seems to be a packaging of Hive, HP Vertica, SparkSQL, Drill and a native ODBC wrapper)
Apache Kylin from Ebay (provides an SQL interface and multi-dimensional analysis [OLAP], "… offers ANSI SQL on Hadoop and supports most ANSI SQL query functions". It depends on HDFS, MapReduce, Hive and HBase; and seems targeted at very large data-sets though maintains low query latency)
Apache Tajo (ANSI/ISO SQL standard compliance with JDBC driver support [benchmarks against Hive and Impala])
Cascading's Lingual² ("Lingual provides JDBC Drivers, a SQL command shell, and a catalog manager for publishing files [or any resource] as schemas and tables.")

Non Open Source, but also interesting
Splice Machine (Standard ANSI SQL, Transactional Integrity)
Pivotal Hawq (via Pivotal HD, ANSI SQL 92, 99 and OLAP)
Cloudera Impala (SQL-like, ANSI 92 compliant, MPP and low-latency)
Impala does not incorporate usage of Hadoop, but leverages the cached data of HDFS on each node to quickly return data (w/o performing Map/Reduce jobs). Thus, overhead related to performing a Map/Reduce job is short-cut and one can gain improvements runtime.
Conclusion: Impala does not replace Hive. However, it is good for different kind of jobs, such as small ad-hoc queries well-suited for analyzing data as business analysts. Robust jobs such as typical ETL taks, on the other hand, require Hive due to the fact that a failure of one job can be very costly.

Thanks to Samuel Marks, who posted originally on the hive user mailing list

Freitag, 9. Januar 2015

Major compact an row key in HBase

Getting an row key via hbase-shell per scan:
hbase (main):0001:0 > scan ‘your_table',{LIMIT => 5}
 ROW
 ....

see what the row contains:
hbase (main):0002:0 > get ‘your_table’,"\x00\x01"
 COLUMN
 ....

To start the compaction based on the row key use this few lines and replace <row_key> and <your_table> with the findings above:
hbase (main):0003:0 > configuration = org.apache.hadoop.hbase.HBaseConfiguration.create table = org.apache.hadoop.hbase.client.HTable.new(configuration, '<your_table>') regionLocation = table.getRegionLocation("<row key>”) regionLocation.getRegionInfo().getRegionName() admin = org.apache.hadoop.hbase.client.HBaseAdmin.new(configuration) admin.majorCompact(regionLocation.getRegionInfo().getRegionName())


Mittwoch, 3. Dezember 2014

The Machine und BigData

HP's Projekt "The Machine" verfolge ich nun schon seit der ersten Veröffentlichung. Bis 2020 soll das Projekt industriereif, ab 2018 sollen erste Edge Devices verfügbar sein. Ob es die Welt der Informatik revolutioniert, bleibt abzuwarten. Extrem interessant ist der Ansatz von HP auf jeden Fall, vor allem im Hinblick auf BigData und die weitere Industrialisierung analytischer Ansätze.

Genutzt wird die Memristor-Technologie (http://en.wikipedia.org/wiki/Memristor). Ein Memristor ist nicht flüchtig, bisher langsamer als DRAM, aber bis zu 100 mal schneller als Flash. Und es kann in relativ kleinen Rackfarmen extrem viel Speicherplatz bereitgestellt werden (4TB haben derzeit den 3.5 Zoll Formfaktor, es könnten aber auch bis zu 100TB per 3.5 Zoll machbar sein).
Das grundlegend neue dabei ist – ein Memristor kann bis zu 10 Zustände speichern (Trinity Memristor). Hierbei wird der Integer auf Basis von 10 berechnet, was im Gegensatz zur herkömmlichen Basis von 8 (64Bit) ein Speicherung auf 20 Bit ermöglicht (bisher 64bit). Folglich hätten die Speicher mit dieser Technologie und bei derzeitiger Fertigungstechnologie die dreifache Kapazität. Ein weiterer interessanter Ansatz ist auch flüchtige Caches in Prozessoren durch nicht flüchtige Speicher zu ersetzen – was wiederum das Computing revolutionieren könnte, etwa durch das Nutzen bereits berechneter Teilmengen in weiteren Verarbeitungsschritten, beispielsweise Mustererkennung in vorhandenen Daten und Übergabe der Muster an einem weiteren Thread zur MCMC Analyse. 

Denkt man an Spark, macht es The Machine durchaus reizvoll – zumal mit Spark vorwiegend nur flüchtige Teilmengen berechnet werden (und bei Bedarf auf ein Speichermedium geschrieben werden, um die berechneten Spills zu persistieren). Und ein solches System wie The Machine würde auch verteilte Dateisysteme wie HDFS oder Ceph größtenteils überflüssig machen, da das Gesamtsystem (also Speicher, RAM, Persistenzlayer, Fast Caching) als ein homogener, nicht flüchtiger Speicherblock funktioniert und jeden bereits berechneten Zustand per se vorrätig halten kann. Diese Teilblöcke könnten dann beliebig wiederverwendet, integriert oder extrapoliert werden, ohne dabei Teile durch volatile caches zu verlieren.

Quellen: