Step by step procedure for HADOOP installation on UBUNTU

Open terminal in UBUNTU and execute the fallowing commands one by one 

 1)sudo apt update

2)sudo apt install openjdk-8-jdk -y

3)java -version; javac -version

4)sudo apt install openssh-server openssh-client -y

5)ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

6)cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

7)chmod 0600 ~/.ssh/authorized_keys

8)ssh localhost

9)wget https://downloads.apache.org/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz

10)tar xzf hadoop-3.2.1.tar.gz

11)sudo nano .bashrc

#Hadoop Related Options
export HADOOP_HOME=/home/hadoop-3.2.1
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_OPTS=”-Djava.library.path=$HADOOP_HOME/lib/native”

12)source ~/.bashrc

13)sudo nano $HADOOP_HOME/etc/hadoop/hadoop-env.sh

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

which javac

readlink -f /usr/bin/javac


14)sudo nano $HADOOP_HOME/etc/hadoop/core-site.xml

<configuration>
<property>
  <name>hadoop.tmp.dir</name>
  <value>/home/tmpdata</value>
</property>
<property>
  <name>fs.default.name</name>
  <value>hdfs://127.0.0.1:9000</value>
</property>
</configuration>

15)sudo nano $HADOOP_HOME/etc/hadoop/hdfs-site.xml

<configuration>
<property>
  <name>dfs.data.dir</name>
  <value>/home/dfsdata/namenode</value>
</property>
<property>
  <name>dfs.data.dir</name>
  <value>/home/dfsdata/datanode</value>
</property>
<property>
  <name>dfs.replication</name>
  <value>1</value>
</property>
</configuration>

16)sudo nano $HADOOP_HOME/etc/hadoop/mapred-site.xml

<configuration>
<property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
</property>
</configuration>


17)sudo nano $HADOOP_HOME/etc/hadoop/yarn-site.xml
<configuration>
<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
</property>
<property>
  <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
  <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
  <name>yarn.resourcemanager.hostname</name>
  <value>127.0.0.1</value>
</property>
<property>
  <name>yarn.acl.enable</name>
  <value>0</value>
</property>
<property>
  <name>yarn.nodemanager.env-whitelist</name>   
  <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PERPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
</property>
</configuration>

18)hdfs namenode -format

Navigate to the hadoop-3.2.1/sbin directory

19)./start-dfs.sh

20)./start-yarn.sh

21)jps

22)http://localhost:9870

23)http://localhost:9864

24)http://localhost:8088


Thank You .....

Comments

  1. i am getting an error after entering this command kindly help.

    in step 11

    hadoop version
    Error: Could not find or load main class ”-Djava.library.path=.home.gurpreet.hadoop-3.3.0.lib.native”

    ReplyDelete
    Replies
    1. Please put the link in double quote

      Delete
    2. same error from my side also but i troubleshoot the error by adding one more line in step 11
      export PATH=$PATH:export PATH=$PATH:/usr/lib/jvm/java-8-openjdk-amd64/bin

      Delete

Post a Comment

Popular posts from this blog

MapReduce Matrix Multiplication Code

Word Count MapReduce Code