Step by step procedure for HADOOP installation on UBUNTU
Open terminal in UBUNTU and execute the fallowing commands one by one
1)sudo apt update
2)sudo apt install openjdk-8-jdk -y
3)java -version; javac -version
4)sudo apt install openssh-server openssh-client -y
5)ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
6)cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
7)chmod 0600 ~/.ssh/authorized_keys
8)ssh localhost
9)wget https://downloads.apache.org/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz
10)tar xzf hadoop-3.2.1.tar.gz
11)sudo nano .bashrc
#Hadoop Related Options
export HADOOP_HOME=/home/hadoop-3.2.1
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_OPTS=”-Djava.library.path=$HADOOP_HOME/lib/native”
12)source ~/.bashrc
13)sudo nano $HADOOP_HOME/etc/hadoop/hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
which javac
readlink -f /usr/bin/javac
14)sudo nano $HADOOP_HOME/etc/hadoop/core-site.xml
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/tmpdata</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://127.0.0.1:9000</value>
</property>
</configuration>
15)sudo nano $HADOOP_HOME/etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>dfs.data.dir</name>
<value>/home/dfsdata/namenode</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/home/dfsdata/datanode</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
16)sudo nano $HADOOP_HOME/etc/hadoop/mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
17)sudo nano $HADOOP_HOME/etc/hadoop/yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>127.0.0.1</value>
</property>
<property>
<name>yarn.acl.enable</name>
<value>0</value>
</property>
<property>
<name>yarn.nodemanager.env-whitelist</name>
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PERPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
</property>
</configuration>
18)hdfs namenode -format
Navigate to the hadoop-3.2.1/sbin directory
19)./start-dfs.sh
20)./start-yarn.sh
21)jps
22)http://localhost:9870
23)http://localhost:9864
24)http://localhost:8088
Thank You .....
i am getting an error after entering this command kindly help.
ReplyDeletein step 11
hadoop version
Error: Could not find or load main class ”-Djava.library.path=.home.gurpreet.hadoop-3.3.0.lib.native”
Please put the link in double quote
Deletesame error from my side also but i troubleshoot the error by adding one more line in step 11
Deleteexport PATH=$PATH:export PATH=$PATH:/usr/lib/jvm/java-8-openjdk-amd64/bin
salesforce training
ReplyDeletehadoop training
Data Science training
linux training
mulesoft training