BDA Lab Internal (April - 2023)

 

SET - I :

a) Implementation of basic Word Count Map Reduce program

b) Create sample HIVE UDF

c) Sqoop import operation

SET - II :

a) Implementation of Matrix Multiplication with Hadoop Map Reduce

b) Create Internal Table and perform DML operations

c) Sqoop eval operation

SET - III :

a) Implementation of Weather mining by taking weather data set using Map Reduce

b) Create External Table and perform DDL operations

c) Sqoop export operation

Comments

Post a Comment

Popular posts from this blog

MapReduce Matrix Multiplication Code

 Mapper Logic  : import java.io.IOException;  import org.apache.hadoop.conf.*;  import org.apache.hadoop.io.*; import org.apache.hadoop.mapreduce.*; public class MatrixMapper extends Mapper<LongWritable, Text, Text, Text>  { public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException  { Configuration conf = context.getConfiguration();  int m = Integer.parseInt(conf.get("m")); int p = Integer.parseInt(conf.get("p"));  String line = value.toString(); String[] indicesAndValue = line.split(",");  Text outputKey = new Text(); Text outputValue = new Text(); if (indicesAndValue[0].equals("M"))  {  for (int k = 0; k < p; k++) { outputKey.set(indicesAndValue[1] + "," + k); outputValue.set("M," + indicesAndValue[2] + "," + indicesAndValue[3]);  context.write(outputKey, outputValue); } }  else  { for (int i = 0; i < m; i++)  { outputKey.set(i + "," + indicesAnd
Read more

Word Count MapReduce Code

  Mapper Logic: import java.io.IOException; import java.util.StringTokenizer; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Mapper; public class WordCountMapper extends Mapper<LongWritable, Text, Text, IntWritable> {   @Override   public void map (LongWritable key, Text value, Context con)       throws IOException, InterruptedException {     String line = value.toString();     String[] words = line.split("\\s");     for(String s:words) {                 con.write(new Text(s), new IntWritable(1));     }   } }     Reducer Logic: import java.io.IOException; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Reducer; public class WordCountReducer extends Reducer<Text, IntWritable, Text, IntWritable> {     @Override   public void reduce (Text key, Iterable&l
Read more

Step by step procedure for HADOOP installation on UBUNTU

Open terminal in UBUNTU and execute the fallowing commands one by one   1)sudo apt update 2)sudo apt install openjdk-8-jdk -y 3)java -version; javac -version 4)sudo apt install openssh-server openssh-client -y 5)ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa 6)cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 7)chmod 0600 ~/.ssh/authorized_keys 8)ssh localhost 9)wget https://downloads.apache.org/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz 10)tar xzf hadoop-3.2.1.tar.gz 11)sudo nano .bashrc #Hadoop Related Options export HADOOP_HOME=/home/hadoop-3.2.1 export HADOOP_INSTALL=$HADOOP_HOME export HADOOP_MAPRED_HOME=$HADOOP_HOME export HADOOP_COMMON_HOME=$HADOOP_HOME export HADOOP_HDFS_HOME=$HADOOP_HOME export YARN_HOME=$HADOOP_HOME export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin export HADOOP_OPTS=”-Djava.library.path=$HADOOP_HOME/lib/native” 12)source ~/.bashrc 13)sudo nano $HADOOP_HOME/etc/hadoop/hadoop-env.sh expor
Read more