Showing posts from February, 2013

Flume 1.3.1 Windows binary release online

Andy Blozhou, a Chinese Flume enthusiast provide precompiled Windows binaries of Flume-1.3.1, including a startup bat and Avro bat.
You can grap this build on their website :

======== snip ========

This is the flume-ng 1.3.1 windows version for download.
simple usage: unzip the run bin/flume.bat for agent.  run bin/flume-avroclient.bat for avro-client.  Need modify for your own env.  detail: (To compile flume-ng on windows, please reference or my chinese version the windows version of flume 1.3.1 file from 2.unzip the to a directory. 3.install jdk 1.6 from oracle,and set JAVA_HOME of the env. download from…

LZO Compression with Oozie

It happens, when one of the compression codec is switched to LZO, that Oozie can't start any MR job successfully. Usually this is done per core-site.xml:

<property> <name>io.compression.codecs</name> <value>,,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec,</value> </property> <property> <name>io.compression.codec.lzo.class</name> <value>com.hadoop.compression.lzo.LzoCodec</value> </property>
Oozie reports a ClassNotFound error (java.lang.ClassNotFoundException: Class com.hadoop.compression.lzo.LzoCodec not found). To get the jobs running copy or link hadoop-lzo.jar into /var/lib/oozie/ and restart Oozie's server. 
The second, most common issue most people forget is to set the shared lib directory:
[root@hadoop2 ~]# sudo -u hdfs hadoop fs -mkdir  /user/oozie [root@hadoop2 …