利用linux搭建测试环境变量,Linux下Hadoop环境配置及测试用例

配置Hadoop环境

下载安装

下载oracle jdk：jdk下载

得到：jdk-14.0.2_linux-x64_bin.tar.gz

下载Hadoop：Hadoop下载

得到：hadoop-3.3.0.tar.gz

将二者移动到/usr/local文件夹，分别解压：

$ sudo tar xzf hadoop-3.3.0.tar.gz

$ sudo mv hadoop-3.3.0 hadoop

$ sudo tar xzf jdk-14.0.2_linux-x64_bin.tar.gz

配置环境变量

$ sudo vim ~/.bashrc

在文件末尾添加：

#set oracle jdk && hadoop environment

export JAVA_HOME=/usr/local/jdk-14.0.2

export HADOOP_HOME=/usr/local/hadoop

export JRE_HOME=${JAVA_HOME}/jre

export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib

export PATH=${JAVA_HOME}/bin:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:$PATH

执行：source ~/.bashrc使配置立即生效。

检查配置是否正确：

$ java -version

java version "14.0.2" 2020-07-14

Java(TM) SE Runtime Environment (build 14.0.2+12-46)

Java HotSpot(TM) 64-Bit Server VM (build 14.0.2+12-46, mixed mode, sharing)

$ hadoop version

Hadoop 3.3.0

Source code repository https://gitbox.apache.org/repos/asf/hadoop.git -r aa96f1871bfd858f9bac59cf2a81ec470da649af

Compiled by brahma on 2020-07-06T18:44Z

Compiled with protoc 3.7.1

From source with checksum 5dc29b802d6ccd77b262ef9d04d19c4

This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-3.3.0.jar

简单测试

利用Hadoop安装提供的示例 MapReduce jar 文件，计算文件的单词总数。

$ mkdir input

$ cp $HADOOP_HOME/*.txt input

$ hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.0.jar wordcount input ouput

output文件夹下出现两个文件：

part-r-00000 _SUCCESS

统计结果存储在 part-r-00000：

"AS 3

"Contribution" 1

"Contributor" 1

"Derivative 1

"Legal 1

"License" 1

"License"); 1

"Licensor" 1

"NOTICE" 1

"Not 1

"Object" 1

"Software"), 1

"Source" 1

"Work" 1

"You" 1

"Your") 1

…

_SUCCESS为空文件，应该是用来指示执行成功的。