Map Reduce 简明教程

MapReduce - Installation

MapReduce仅在类Linux的操作系统上运行，并内置于Hadoop框架中。我们需要执行以下步骤来安装Hadoop框架。

MapReduce works only on Linux flavored operating systems and it comes inbuilt with a Hadoop Framework. We need to perform the following steps in order to install Hadoop framework.

Verifying JAVA Installation

在安装Hadoop之前，必须在系统中安装Java。使用以下命令检查您的系统是否已安装Java。

Java must be installed on your system before installing Hadoop. Use the following command to check whether you have Java installed on your system.

$ java –version

如果系统上已安装 Java，您将看到以下响应 -

If Java is already installed on your system, you get to see the following response −

java version "1.7.0_71"
Java(TM) SE Runtime Environment (build 1.7.0_71-b13)
Java HotSpot(TM) Client VM (build 25.0-b02, mixed mode)

如果您尚未在系统中安装Java，请按照以下步骤操作。

In case you don’t have Java installed on your system, then follow the steps given below.

Installing Java

Step 1

从以下链接下载Java最新版本： this link 。

Download the latest version of Java from the following link − this link.

下载后，您可以在“下载”文件夹中找到文件 jdk-7u71-linux-x64.tar.gz 。

After downloading, you can locate the file jdk-7u71-linux-x64.tar.gz in your Downloads folder.

Step 2

使用以下命令来解压jdk-7u71-linux-x64.gz的内容。

Use the following commands to extract the contents of jdk-7u71-linux-x64.gz.

$ cd Downloads/
$ ls
jdk-7u71-linux-x64.gz
$ tar zxf jdk-7u71-linux-x64.gz
$ ls
jdk1.7.0_71 jdk-7u71-linux-x64.gz

Step 3

要向所有用户提供Java，您必须将其移动到“/usr/local/”位置。转到root并键入以下命令：

To make Java available to all the users, you have to move it to the location “/usr/local/”. Go to root and type the following commands −

$ su
password:
# mv jdk1.7.0_71 /usr/local/java
# exit

Step 4

为设置 PATH 和 JAVA_HOME 变量，将以下命令添加到 ~/.bashrc 文件中。

For setting up PATH and JAVA_HOME variables, add the following commands to ~/.bashrc file.

export JAVA_HOME=/usr/local/java
export PATH=$PATH:$JAVA_HOME/bin

将所有更改应用到当前运行的系统。

Apply all the changes to the current running system.

$ source ~/.bashrc

Step 5

使用以下命令配置Java备用：

Use the following commands to configure Java alternatives −

# alternatives --install /usr/bin/java java usr/local/java/bin/java 2

# alternatives --install /usr/bin/javac javac usr/local/java/bin/javac 2

# alternatives --install /usr/bin/jar jar usr/local/java/bin/jar 2

# alternatives --set java usr/local/java/bin/java

# alternatives --set javac usr/local/java/bin/javac

# alternatives --set jar usr/local/java/bin/jar

现在使用命令 java -version 从终端验证安装。

Now verify the installation using the command java -version from the terminal.

Verifying Hadoop Installation

在安装MapReduce之前，必须在您的系统上安装Hadoop。让我们使用以下命令验证Hadoop安装：

Hadoop must be installed on your system before installing MapReduce. Let us verify the Hadoop installation using the following command −

$ hadoop version

如果已在您的系统上安装 Hadoop，则会收到以下回复：

If Hadoop is already installed on your system, then you will get the following response −

Hadoop 2.4.1
--
Subversion https://svn.apache.org/repos/asf/hadoop/common -r 1529768
Compiled by hortonmu on 2013-10-07T06:28Z
Compiled with protoc 2.5.0
From source with checksum 79e53ce7994d1628b240f09af91e1af4

如果您的系统上尚未安装Hadoop，请继续执行以下步骤。

If Hadoop is not installed on your system, then proceed with the following steps.

Downloading Hadoop

从Apache软件基金会下载Hadoop 2.4.1，并使用以下命令解压其内容。

Download Hadoop 2.4.1 from Apache Software Foundation and extract its contents using the following commands.

$ su
password:
# cd /usr/local
# wget http://apache.claz.org/hadoop/common/hadoop-2.4.1/
hadoop-2.4.1.tar.gz
# tar xzf hadoop-2.4.1.tar.gz
# mv hadoop-2.4.1/* to hadoop/
# exit

Installing Hadoop in Pseudo Distributed mode

以下步骤用于在伪分布模式下安装 Hadoop 2.4.1。

The following steps are used to install Hadoop 2.4.1 in pseudo distributed mode.

Step 1 − Setting up Hadoop

您可以通过将以下命令附加到~/.bashrc文件来设置Hadoop环境变量。

You can set Hadoop environment variables by appending the following commands to ~/.bashrc file.

export HADOOP_HOME=/usr/local/hadoop
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

将所有更改应用到当前运行的系统。

Apply all the changes to the current running system.

$ source ~/.bashrc

Step 2 − Hadoop Configuration

您可以在位置 “$HADOOP_HOME/etc/hadoop” 中找到所有 Hadoop 配置文件。根据您的 Hadoop 基础架构，您需要在这些配置文件中进行适当的更改。

You can find all the Hadoop configuration files in the location “$HADOOP_HOME/etc/hadoop”. You need to make suitable changes in those configuration files according to your Hadoop infrastructure.

$ cd $HADOOP_HOME/etc/hadoop

为了使用 Java 开发 Hadoop 程序，您必须替换 hadoop-env.sh 文件中的 JAVA_HOME 值为系统中 Java 的位置，以重置 Java 环境变量。

In order to develop Hadoop programs using Java, you have to reset the Java environment variables in hadoop-env.sh file by replacing JAVA_HOME value with the location of Java in your system.

export JAVA_HOME=/usr/local/java

您必须编辑以下文件以配置 Hadoop -

You have to edit the following files to configure Hadoop −

core-site.xml
hdfs-site.xml
yarn-site.xml
mapred-site.xml

core-site.xml

core-site.xml 包含以下信息 -

core-site.xml contains the following information−

Port number used for Hadoop instance
Memory allocated for the file system
Memory limit for storing the data
Size of Read/Write buffers

打开 core-site.xml，并在 <configuration> 和 </configuration> 标记之间添加以下属性。

Open the core-site.xml and add the following properties in between the <configuration> and </configuration> tags.

<configuration>
   <property>
      <name>fs.default.name</name>
      <value>hdfs://localhost:9000 </value>
   </property>
</configuration>

hdfs-site.xml

hdfs-site.xml 包含以下信息 -

hdfs-site.xml contains the following information −

Value of replication data
The namenode path
The datanode path of your local file systems (the place where you want to store the Hadoop infra)

让我们假设以下数据。

Let us assume the following data.

dfs.replication (data replication value) = 1

(In the following path /hadoop/ is the user name.
hadoopinfra/hdfs/namenode is the directory created by hdfs file system.)
namenode path = //home/hadoop/hadoopinfra/hdfs/namenode

(hadoopinfra/hdfs/datanode is the directory created by hdfs file system.)
datanode path = //home/hadoop/hadoopinfra/hdfs/datanode

打开该文件，并在 <configuration>、</configuration> 标记之间添加以下属性。

Open this file and add the following properties in between the <configuration>, </configuration> tags.

<configuration>

   <property>
      <name>dfs.replication</name>
      <value>1</value>
   </property>

   <property>
      <name>dfs.name.dir</name>
      <value>file:///home/hadoop/hadoopinfra/hdfs/namenode</value>
   </property>

   <property>
      <name>dfs.data.dir</name>
      <value>file:///home/hadoop/hadoopinfra/hdfs/datanode </value>
   </property>

</configuration>

Note − 在上述文件中，所有属性值都是用户定义的，您可以根据 Hadoop 基础架构进行更改。

Note − In the above file, all the property values are user-defined and you can make changes according to your Hadoop infrastructure.

yarn-site.xml

此文件用于将纱线配置到 Hadoop 中。打开 yarn-site.xml 文件并在 <configuration>，</configuration> 标记之间添加以下属性。

This file is used to configure yarn into Hadoop. Open the yarn-site.xml file and add the following properties in between the <configuration>, </configuration> tags.

<configuration>
   <property>
      <name>yarn.nodemanager.aux-services</name>
      <value>mapreduce_shuffle</value>
   </property>
</configuration>

mapred-site.xml

此文件用于指定我们正在使用的 MapReduce 框架。默认情况下，Hadoop 包含 yarn-site.xml 的模板。首先，您需要使用以下命令将文件从 mapred-site.xml.template 复制到 mapred-site.xml 文件。

This file is used to specify the MapReduce framework we are using. By default, Hadoop contains a template of yarn-site.xml. First of all, you need to copy the file from mapred-site.xml.template to mapred-site.xml file using the following command.

$ cp mapred-site.xml.template mapred-site.xml

打开 mapred-site.xml 文件并在 <configuration>，</configuration> 标记中添加以下属性。

Open mapred-site.xml file and add the following properties in between the <configuration>, </configuration> tags.

<configuration>
   <property>
      <name>mapreduce.framework.name</name>
      <value>yarn</value>
   </property>
</configuration>

Verifying Hadoop Installation

以下步骤用于验证 Hadoop 安装。

The following steps are used to verify the Hadoop installation.

Step 1 − Name Node Setup

使用 “hdfs namenode -format” 命令设置 name 节点，如下所示：

Set up the namenode using the command “hdfs namenode -format” as follows −

$ cd ~
$ hdfs namenode -format

预期结果如下：

The expected result is as follows −

10/24/14 21:30:55 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = localhost/192.168.1.11
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.4.1
...
...
10/24/14 21:30:56 INFO common.Storage: Storage directory
/home/hadoop/hadoopinfra/hdfs/namenode has been successfully formatted.
10/24/14 21:30:56 INFO namenode.NNStorageRetentionManager: Going to
retain 1 images with txid >= 0
10/24/14 21:30:56 INFO util.ExitUtil: Exiting with status 0
10/24/14 21:30:56 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localhost/192.168.1.11
************************************************************/

Step 2 − Verifying Hadoop dfs

执行以下命令以启动您的 Hadoop 文件系统。

Execute the following command to start your Hadoop file system.

$ start-dfs.sh

预期输出如下所示 −

The expected output is as follows −

10/24/14 21:37:56
Starting namenodes on [localhost]
localhost: starting namenode, logging to /home/hadoop/hadoop-
2.4.1/logs/hadoop-hadoop-namenode-localhost.out
localhost: starting datanode, logging to /home/hadoop/hadoop-
2.4.1/logs/hadoop-hadoop-datanode-localhost.out
Starting secondary namenodes [0.0.0.0]

Step 3 − Verifying Yarn Script

以下命令用于启动 Yarn 脚本。执行此命令将启动您的 Yarn 守护程序。

The following command is used to start the yarn script. Executing this command will start your yarn daemons.

$ start-yarn.sh

预期输出如下所示 −

The expected output is as follows −

starting yarn daemons
starting resourcemanager, logging to /home/hadoop/hadoop-
2.4.1/logs/yarn-hadoop-resourcemanager-localhost.out
localhost: starting node manager, logging to /home/hadoop/hadoop-
2.4.1/logs/yarn-hadoop-nodemanager-localhost.out

Step 4 − Accessing Hadoop on Browser

访问 Hadoop 的默认端口号是 50070。使用以下 URL 在您的浏览器上获取 Hadoop 服务。

The default port number to access Hadoop is 50070. Use the following URL to get Hadoop services on your browser.

http://localhost:50070/

以下屏幕截图显示了 Hadoop 浏览器。

The following screenshot shows the Hadoop browser.

Step 5 − Verify all Applications of a Cluster

访问群集中所有应用程序的默认端口号为 8088。使用以下 URL 使用此服务。

The default port number to access all the applications of a cluster is 8088. Use the following URL to use this service.

http://localhost:8088/

以下屏幕截图显示了 Hadoop 群集浏览器。

The following screenshot shows a Hadoop cluster browser.