{"id":256,"date":"2012-05-17T09:12:39","date_gmt":"2012-05-17T13:12:39","guid":{"rendered":"http:\/\/digitallibraryworld.com\/?p=256"},"modified":"2012-05-18T08:14:03","modified_gmt":"2012-05-18T12:14:03","slug":"installing-hadoop-on-debian-a-more-detailed-guide","status":"publish","type":"post","link":"https:\/\/heisbudi.com\/?p=256","title":{"rendered":"Installing Hadoop on Debian &#8211; a more detailed guide"},"content":{"rendered":"<p style=\"padding-left: 30px;\">I&#8217;m hooked on Debian as of late. I&#8217;m still kinda new to Linux environment. I&#8217;ve been on Windows side for waaay too long. So, after familiarizing myself with Linux for the past 2 months, I decided to pick up a book about <a title=\"Hadoop by Tom White\" href=\"http:\/\/www.amazon.com\/Hadoop-Definitive-Guide-Tom-White\/dp\/1449389732\/ref=sr_1_2?ie=UTF8&amp;qid=1336739469&amp;sr=8-2\" target=\"_blank\">Hadoop<\/a>, mainly because I&#8217;m interested in processing big data. While this is a great book, it seems to assume that you are familiar with Linux and Java. This has been a fun learning experience for me. This might be useful for others who might be struggling to get Hadoop set up for the first time. If you are a Debian guru, please be gentle. This is my first Debian related post.<\/p>\n<p style=\"padding-left: 30px;\">Let&#8217;s dive in. I&#8217;m assuming that you have a clean install of Debian, with nothing but SSH installed. You need to have the following package installed:<\/p>\n<ul style=\"padding-left: 30px;\">\n<li>sudo(optional). To install, login as root(type <strong>su<\/strong>, enter your password), run\u00a0<strong>apt-get install sudo.<\/strong>\n<ul>\n<li>give your username the ability to sudo by adding the following line to <span style=\"text-decoration: underline;\">\/etc\/sudoers<\/span>\n<ul>\n<li><strong>vi \/etc\/sudoers<\/strong>, and add the following line under <em>User Privelege Secifications<\/em>(hit <strong>i <\/strong>key to insert text, and <strong>escape<\/strong> key to get out of the edit\/insert mode. Type in <strong>:wq <\/strong>to save a file and quit the editor<\/li>\n<li><em>yourUserName ALL=(ALL) ALL<\/em><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li>vim (type <strong>sudo <\/strong><strong>apt-get install vim<\/strong>). You also need an SSH server, which I installed during my Debian installation.<\/li>\n<li>Generate private and public key pair for the current user:<\/li>\n<li>type <strong>ssh-keygen<\/strong> and accept the default location by hitting enter.<\/li>\n<li>You can choose to protect your private key with a password.<\/li>\n<li>After the pair is generated, run <strong>cat ~\/.ssh\/id_rsa.pub &gt;&gt; ~\/.ssh\/authorized_keys<\/strong><\/li>\n<li>To make sure that this is done correctly, run <strong>ssh localhost<\/strong>. You should get a prompt, without having to type in password again<\/li>\n<li>The authenticity of host &#8216;localhost (127.0.0.1)&#8217; can&#8217;t be established.<br \/>\n<span style=\"font-style: italic;\">RSA key fingerprint is xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx.<br \/>\n<\/span><span style=\"font-style: italic;\">Are you sure you want to continue connecting (yes\/no)? <strong>yes<\/strong><br \/>\n<\/span><span style=\"font-style: italic;\">Warning: Permanently added &#8216;localhost&#8217; (RSA) to the list of known hosts.<br \/>\n<\/span><em>The authenticity of host &#8216;localhost (127.0.0.1)&#8217; can&#8217;t be established.RSA key fingerprint is .<br \/>\n<\/em>The next time you ssh in, the message above shouldn&#8217;t appear again.<\/li>\n<\/ul>\n<p>All those above are debian configuration. Now, let&#8217;s try to set up single-node Hadoop. Some of this is described in Appendix A of the book I mentioned above. However, the instruction seems to oversimplify stuff. I&#8217;ll try to go into more details on how to install Java and Hadoop for a first timer. If you just follow the installation instruction on Appendix A, and try to run the command on page 23:<\/p>\n<pre><strong>$ export HADOOP_CLASSPATH=build\/classes\r\n<\/strong><strong>$ hadoop MaxTemperature input\/ncdc\/sample.txt output<\/strong><\/pre>\n<p>You will get the following error(even after installing JDK):<\/p>\n<p><span style=\"color: #ff0000;\">Exception in thread &#8220;main&#8221; java.lang.NoClassDefFoundError: MaxTemperature<br \/>\nCaused by: java.lang.ClassNotFoundException: MaxTemperature<\/span><span style=\"color: #ff0000;\"> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)<br \/>\nat java.security.AccessController.doPrivileged(Native Method)<br \/>\nat java.net.URLClassLoader.findClass(URLClassLoader.java:190)<br \/>\nat java.lang.ClassLoader.loadClass(ClassLoader.java:306)<br \/>\nat sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)<br \/>\nat java.lang.ClassLoader.loadClass(ClassLoader.java:247)<!--more--><\/span><\/p>\n<p>If you know how Java works, this is because, you still haven&#8217;t compiled your .java files into .class files for JVM to run. That&#8217;s what the error above is saying. You need to compile all your *.java files into *.class files. In order to do that, you need to install and configure JDK and HADOOP installation.<\/p>\n<ul style=\"padding-left: 30px;\">\n<li>First thing you need is JDK &#8211; Java Development Kit. To get this, you need to update your <strong>sources.list <\/strong>file(a source repository where you get all your package for your debian). This package is listed under debian non free package.<\/li>\n<li>Type the following command at the prompt:<span style=\"font-weight: bold;\">sudo vim \/etc\/apt\/sources.list<\/span><\/li>\n<li>Add the following to your <strong>sources.list<\/strong>. I&#8217;m using repository hosted by Indiana University. If you prefer a repository closer to your location, you can look it up <a title=\"List of Debian repository mirrors\" href=\"http:\/\/www.debian.org\/mirror\/list\" target=\"_blank\">here<\/a>:<\/li>\n<\/ul>\n<pre style=\"padding-left: 60px;\"><em>#JAVA non free\r\ndeb http:\/\/ftp.uwsg.indiana.edu\/linux\/debian\/ squeeze main non-free\r\ndeb-src http:\/\/ftp.uwsg.indiana.edu\/linux\/debian\/ squeeze main non-free<\/em><\/pre>\n<ul>\n<li>While you are at it, add the following repository too so that you can install Hadoop later:<\/li>\n<\/ul>\n<p style=\"padding-left: 60px;\"><em>#HADOOP<br \/>\ndeb http:\/\/archive.cloudera.com\/debian squeeze-cdh3 contrib<br \/>\ndeb-src http:\/\/archive.cloudera.com\/debian squeeze-cdh3 contrib<\/em><\/p>\n<ul>\n<li>run <strong>sudo apt-get update<\/strong><\/li>\n<li>install java by running <strong>sudo apt-get install sun-java6-jdk<\/strong>. Confirm the license agreement. This should download and install JDK from the repository you specified above<\/li>\n<li>Now, add CloudEra.com&#8217;s public key to your system. This is so that you can make sure packages are trusted(that they are from cloudera indeed). From your home directory(<em>user@host:~$<\/em> prompt)Run\u00a0<strong>wget http:\/\/archive.cloudera.com\/debian\/cloudera.key &amp;&amp; sudo apt-key add ~\/cloudera.key. <\/strong>To see if cloud era&#8217;s key is added, run <strong>sudo apt-key list.<\/strong> If the key is added correctly, you should see an entry.<\/li>\n<li>Now, install hadoop by running\u00a0\u00a0<strong>sudo apt-get install\u00a0hadoop-0.20 hadoop-0.20-namenode hadoop-0.20-datanode hadoop-0.20-jobtracker hadoop-0.20-tasktracker. <\/strong> If you didn&#8217;t add cloud era&#8217;s public key, you will be warned that the source of the package is not authenticated.<\/li>\n<li>Now it&#8217;s time to setup several environment variables:<\/li>\n<\/ul>\n<ol>\n<li>after the installation, there should be a directory called <em>hadoop-0.20 <\/em>within <em>\/usr\/lib\/ <\/em>directory<em>. <\/em>This is the value of HADOOP_INSTALL environment variable\u00a0To check, run <strong>ls -l -d h \/usr\/lib\/h*\/.<\/strong><strong> <\/strong><strong><span style=\"text-decoration: underline;\">Make note of this value<strong>!!<\/strong><\/span><\/strong><\/li>\n<li>get the path to your java install location by running\u00a0<strong>readlink -f \/usr\/bin\/java | sed &#8220;s:\/bin\/java::&#8221;. <\/strong>This will follow symbolic link from \/usr\/bin\/java all the way down to its actual folder location. Copy the result of this command! This is your JAVA_HOME value.<\/li>\n<li>Run this command, and make note of the result:\u00a0<strong>hadoop version | sed &#8220;s:^Hadoop 0:hadoop-0:&#8221; |head\u00a0-n 1<\/strong>. This is the value of your HADOOP_VERSION.<\/li>\n<li>I like things organized. So, I created the following directory structure in my home directory:<\/li>\n<li>~\/hadoop\/build\/sources \u00a0&#8211; This where I store all my *.java files<\/li>\n<li>~\/hadoop\/build\/classes &#8211; This is where I store all my *.classes files(The output after compiling *.java files using <strong>javac<\/strong> command)<\/li>\n<li>~\/hadoop\/input &#8211; This is where I store all the text files to be processed by MapReduce<\/li>\n<li>~\/hadoop\/output &#8211; This is where I store all the output after being processed by MapReduce<\/li>\n<\/ol>\n<ul>\n<li>Run <strong>vim ~\/.bashrc<\/strong><strong>, <\/strong><strong>add the following at the very end(hit <strong>G<\/strong> and <strong>o<\/strong> if you are in VIM). You need to adjust it based on the output of #1,2,3 above, and also how you organize your \u00a0#4. hashmark (#) indicates a comment.<\/strong><\/li>\n<\/ul>\n<pre style=\"padding-left: 60px;\"><span style=\"font-style: italic;\"># for hadoop and java setting\r\n<\/span><span style=\"font-style: italic;\">#based on #6 above.\r\n<\/span><span style=\"font-weight: bold;\">declare -x HADOOP_CLASSPATH=~\/hadoop\/build\/classes\r\n<\/span><span style=\"font-style: italic;\">#based on # 1 above\r\n<\/span><span style=\"font-weight: bold;\">declare -x HADOOP_INSTALL=\"\/usr\/lib\/hadoop-0.20\"\r\n<\/span><span style=\"font-style: italic;\">#based on # 3 above\r\n<\/span><span style=\"font-weight: bold;\">declare -x HADOOP_VERSION=\"hadoop-0.20.2-cdh3u3\"\r\n<\/span><span style=\"font-style: italic;\">#based on # 2 above\r\n<\/span><span style=\"font-weight: bold;\">declare -x JAVA_HOME=\"\/usr\/lib\/jvm\/java-6-sun-1.6.0.26\/jre\"\r\n<\/span><span style=\"font-style: italic;\">#to tell java where to look for classes or library\r\n<\/span><span style=\"font-weight: bold;\">declare -x CLASSPATH=$HADOOP_INSTALL\/$HADOOP_VERSION-core.jar:~\/hadoop\/build\/classes\r\n<\/span><span style=\"font-weight: bold;\">export PATH=$PATH:\/sbin:$JAVA_HOME\/bin\r\n<\/span><\/pre>\n<ul>\n<li>escape from the insert mode, and save the file and quit(hit <strong>:wq<\/strong> in VIM)run <strong>bash<\/strong> command. This would apply the profile you just added above.<\/li>\n<li>Now you are ready to compile all your *.java files. I&#8217;m assuming you have all the *.java files stored in ~\/hadoop\/build\/sources. I will be storing the compiled code in ~\/hadoop\/build\/classes. Run the following command to compile all MaxTemperatureReducer.java, MaxTemperatureMapper.java and MaxTemperatureReducer.java\n<ul>\n<li><strong>javac ~\/hadoop\/build\/sources\/*.java -d ~\/hadoop\/build\/classes<\/strong><\/li>\n<li>If all goes well, all your *.java files will be compiled, and the resulting <em>*.class<\/em> files will be store in <em>~\/hadoop\/build\/classes<\/em>.\u00a0Check your classes directory by running: <strong>ls ~\/hadoop\/build\/classes. <\/strong>You should see 3 corresponding class files.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<ul>\n<li>Now we are ready to run hadoop. Make sure you have an appropriate input file expected by the compiled application. If you don&#8217;t, you can download it from\u00a0<a title=\"Weather data set\" href=\"http:\/\/ftp3.ncdc.noaa.gov\/pub\/data\/noaa\/\" target=\"_blank\">http:\/\/ftp3.ncdc.noaa.gov\/pub\/data\/noaa\/<\/a>. You can download the data by running wget. For example, to download a file, <span style=\"color: #0000ff;\">http:\/\/ftp3.ncdc.noaa.gov\/pub\/data\/noaa\/2012\/010010-99999-2012.gz<\/span> to \u00a0your ~\/hadoop\/input\/ncdc\/ folder, run\u00a0<strong>wget -P ~\/hadoop\/input\/ncdc\/ http:\/\/ftp3.ncdc.noaa.gov\/pub\/data\/noaa\/1911\/029170-99999-1911.gz<\/strong>. To unzip this file into a plain text file, run <strong>gunzip -d ~\/hadoop\/input\/ncdc\/*.gz. <\/strong>You can rename this file to sample.txt by running <strong>mv ~\/hadoop\/input\/ncdc\/029170-99999-1911 ~\/hadoop\/input\/ncdc\/sample.txt<\/strong><\/li>\n<\/ul>\n<ul>\n<li>Try running hadoop again as instructed by the book. We don&#8217;t have to worry about setting the HADOOP_CLASSPATH now \u00a0because we already set it as environment variable. Run\u00a0 <span style=\"font-weight: bold;\">hadoop MaxTemperature ~\/hadoop\/input\/ncdc\/sample.txt \u00a0~hadoop\/output. <\/span><\/li>\n<\/ul>\n<p>If all is well, you should see output similar what&#8217;s printed in the book. This is the output:<\/p>\n<p>12\/05\/14 12:07:26 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=<br \/>\n12\/05\/14 12:07:26 INFO util.NativeCodeLoader: Loaded the native-hadoop library<br \/>\n12\/05\/14 12:07:26 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.<br \/>\n12\/05\/14 12:07:26 WARN mapred.JobClient: No job jar file set. \u00a0User classes may\u00a0not be found. See JobConf(Class) or JobConf#setJar(String). \u00a0 \u00a0 \u00a0 \u00a012\/05\/14 12:07:27 WARN snappy.LoadSnappy: Snappy native library is available<br \/>\n12\/05\/14 12:07:27 INFO snappy.LoadSnappy: Snappy native library loaded<br \/>\n12\/05\/14 12:07:27 INFO mapred.FileInputFormat: Total input paths to process : 1<br \/>\n12\/05\/14 12:07:27 INFO mapred.JobClient: Running job: job_local_0001<br \/>\n12\/05\/14 12:07:28 INFO util.ProcessTree: setsid exited with exit code 0<br \/>\n12\/05\/14 12:07:28 INFO mapred.Task: \u00a0Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@10d09ad3 \u00a0 \u00a0 \u00a0\u00a012\/05\/14 12:07:28 INFO mapred.MapTask: numReduceTasks: 1<br \/>\n12\/05\/14 12:07:28 INFO mapred.MapTask: io.sort.mb = 100<br \/>\n12\/05\/14 12:07:28 INFO mapred.MapTask: data buffer = 79691776\/99614720<br \/>\n12\/05\/14 12:07:28 INFO mapred.MapTask: record buffer = 262144\/327680<br \/>\n12\/05\/14 12:07:28 INFO mapred.MapTask: Starting flush of map output<br \/>\n12\/05\/14 12:07:28 INFO mapred.MapTask: Finished spill 0<br \/>\n12\/05\/14 12:07:28 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done.\u00a0And is in the process of commiting \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a012\/05\/14 12:07:28 INFO mapred.LocalJobRunner: file:\/home\/bwibowo\/hadoop\/input\/ncdc\/sample.txt:0+150522 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0\u00a012\/05\/14 12:07:28 INFO mapred.Task: Task &#8216;attempt_local_0001_m_000000_0&#8217; done.<br \/>\n12\/05\/14 12:07:28 INFO mapred.Task: \u00a0Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3c50507 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0\u00a012\/05\/14 12:07:28 INFO mapred.LocalJobRunner:<br \/>\n12\/05\/14 12:07:28 INFO mapred.Merger: Merging 1 sorted segments<br \/>\n12\/05\/14 12:07:28 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 11937 bytes<br \/>\n12\/05\/14 12:07:28 INFO mapred.LocalJobRunner:<br \/>\n12\/05\/14 12:07:28 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is done.\u00a0And is in the process of commiting<br \/>\n12\/05\/14 12:07:28 INFO mapred.LocalJobRunner:<br \/>\n12\/05\/14 12:07:28 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is allowed to commit now<\/p>\n<div data-counters='1' data-style='square' data-size='regular' data-url='https:\/\/heisbudi.com\/?p=256' data-title='Installing Hadoop on Debian &#8211; a more detailed guide' class='linksalpha_container linksalpha_app_3'><a href='\/\/www.linksalpha.com\/share?network='facebook' class='linksalpha_icon_facebook'><\/a><a href='\/\/www.linksalpha.com\/share?network='twitter' class='linksalpha_icon_twitter'><\/a><a href='\/\/www.linksalpha.com\/share?network='googleplus' class='linksalpha_icon_googleplus'><\/a><a href='\/\/www.linksalpha.com\/share?network='mail' class='linksalpha_icon_mail'><\/a><\/div><div data-position='' data-url='https:\/\/heisbudi.com\/?p=256' data-title='Installing Hadoop on Debian &#8211; a more detailed guide' class='linksalpha_container linksalpha_app_7'><a href='\/\/www.linksalpha.com\/share?network='facebook' class='linksalpha_icon_facebook'><\/a><a href='\/\/www.linksalpha.com\/share?network='twitter' class='linksalpha_icon_twitter'><\/a><a href='\/\/www.linksalpha.com\/share?network='googleplus' class='linksalpha_icon_googleplus'><\/a><a href='\/\/www.linksalpha.com\/share?network='mail' class='linksalpha_icon_mail'><\/a><\/div>","protected":false},"excerpt":{"rendered":"<p>Installing Hadoop on Debian &#8211; a more detailed guide from O&#8217;Reilly book<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[23,22],"tags":[],"_links":{"self":[{"href":"https:\/\/heisbudi.com\/index.php?rest_route=\/wp\/v2\/posts\/256"}],"collection":[{"href":"https:\/\/heisbudi.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/heisbudi.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/heisbudi.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/heisbudi.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=256"}],"version-history":[{"count":34,"href":"https:\/\/heisbudi.com\/index.php?rest_route=\/wp\/v2\/posts\/256\/revisions"}],"predecessor-version":[{"id":290,"href":"https:\/\/heisbudi.com\/index.php?rest_route=\/wp\/v2\/posts\/256\/revisions\/290"}],"wp:attachment":[{"href":"https:\/\/heisbudi.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=256"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/heisbudi.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=256"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/heisbudi.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=256"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}