Apache Spark 简明教程

Apache Spark - Introduction

各行业都在广泛使用 Hadoop 来分析其数据集。原因在于 Hadoop 框架基于一个简单的编程模型（MapReduce），并且它提供了一个可扩展、灵活、容错且具有成本效益的计算解决方案。这里，主要顾虑是在处理大型数据集的速度，即查询之间的等待时间和运行程序的等待时间。

Industries are using Hadoop extensively to analyze their data sets. The reason is that Hadoop framework is based on a simple programming model (MapReduce) and it enables a computing solution that is scalable, flexible, fault-tolerant and cost effective. Here, the main concern is to maintain speed in processing large datasets in terms of waiting time between queries and waiting time to run the program.

Apache 软件基金会推出了 Spark 来加速 Hadoop 计算软件流程。

Spark was introduced by Apache Software Foundation for speeding up the Hadoop computational computing software process.

与普遍的看法相反， Spark is not a modified version of Hadoop 并不真正依赖于 Hadoop，因为它有自己的集群管理。Hadoop 只是实现 Spark 的方式之一。

As against a common belief, Spark is not a modified version of Hadoop and is not, really, dependent on Hadoop because it has its own cluster management. Hadoop is just one of the ways to implement Spark.

Spark 以两种方式使用 Hadoop - 一种是 storage ，第二种是 processing 。由于 Spark 有自己的集群管理计算，所以它只将 Hadoop 用于存储目的。

Spark uses Hadoop in two ways – one is storage and second is processing. Since Spark has its own cluster management computation, it uses Hadoop for storage purpose only.

Apache Spark

Apache Spark 是一种闪电般快速的集群计算技术，专为快速计算而设计。它基于 Hadoop MapReduce，并扩展了 MapReduce 模型以有效地将其用于更多类型的计算，包括交互式查询和流处理。Spark 的主要特征是其 in-memory cluster computing ，它提高了应用程序的处理速度。

Apache Spark is a lightning-fast cluster computing technology, designed for fast computation. It is based on Hadoop MapReduce and it extends the MapReduce model to efficiently use it for more types of computations, which includes interactive queries and stream processing. The main feature of Spark is its in-memory cluster computing that increases the processing speed of an application.

Spark 旨在涵盖广泛的工作负载，例如批处理应用程序、迭代算法、交互式查询和流处理。除了在各自系统中支持所有这些工作负载外，它还减轻了维护单独工具的管理负担。

Spark is designed to cover a wide range of workloads such as batch applications, iterative algorithms, interactive queries and streaming. Apart from supporting all these workload in a respective system, it reduces the management burden of maintaining separate tools.

Evolution of Apache Spark

Spark 是 Hadoop 的一个子项目，于 2009 年在加州大学伯克利分校 AMPLab 由 Matei Zaharia 开发。它于 2010 年根据 BSD 许可证开源。它于 2013 年捐赠给 Apache 软件基金会，现在 Apache Spark 已成为自 2014 年 2 月起的一个顶级 Apache 项目。

Spark is one of Hadoop’s sub project developed in 2009 in UC Berkeley’s AMPLab by Matei Zaharia. It was Open Sourced in 2010 under a BSD license. It was donated to Apache software foundation in 2013, and now Apache Spark has become a top level Apache project from Feb-2014.

Features of Apache Spark

Apache Spark 具有以下功能。

Apache Spark has following features.

  1. Speed − Spark helps to run an application in Hadoop cluster, up to 100 times faster in memory, and 10 times faster when running on disk. This is possible by reducing number of read/write operations to disk. It stores the intermediate processing data in memory.

  2. Supports multiple languages − Spark provides built-in APIs in Java, Scala, or Python. Therefore, you can write applications in different languages. Spark comes up with 80 high-level operators for interactive querying.

  3. Advanced Analytics − Spark not only supports ‘Map’ and ‘reduce’. It also supports SQL queries, Streaming data, Machine learning (ML), and Graph algorithms.

Spark Built on Hadoop

下图展示了使用 Hadoop 组件构建 Spark 的三种方法。

The following diagram shows three ways of how Spark can be built with Hadoop components.

spark built on hadoop

如以下说明所示，共有三种 Spark 部署方法。

There are three ways of Spark deployment as explained below.

  1. Standalone − Spark Standalone deployment means Spark occupies the place on top of HDFS(Hadoop Distributed File System) and space is allocated for HDFS, explicitly. Here, Spark and MapReduce will run side by side to cover all spark jobs on cluster.

  2. Hadoop Yarn − Hadoop Yarn deployment means, simply, spark runs on Yarn without any pre-installation or root access required. It helps to integrate Spark into Hadoop ecosystem or Hadoop stack. It allows other components to run on top of stack.

  3. Spark in MapReduce (SIMR) − Spark in MapReduce is used to launch spark job in addition to standalone deployment. With SIMR, user can start Spark and uses its shell without any administrative access.

Components of Spark

下图描绘了 Spark 的不同组件。

The following illustration depicts the different components of Spark.

components of spark

Apache Spark Core

Spark 核心是 Spark 平台的基础通用执行引擎，所有其他功能都建立在其之上。它提供内存计算和外部存储系统中的引用数据集。

Spark Core is the underlying general execution engine for spark platform that all other functionality is built upon. It provides In-Memory computing and referencing datasets in external storage systems.

Spark SQL

Spark SQL 是 Spark 核心之上的一个组件，它引入了名为 SchemaRDD 的新数据抽象，该抽象为结构化和半结构化数据提供了支持。

Spark SQL is a component on top of Spark Core that introduces a new data abstraction called SchemaRDD, which provides support for structured and semi-structured data.

Spark Streaming

Spark Streaming 利用 Spark 核心 的快速调度功能执行流分析。它以小批量的方式获取数据，并在这些数据小批量上执行 RDD（弹性分布式数据集）转换。

Spark Streaming leverages Spark Core’s fast scheduling capability to perform streaming analytics. It ingests data in mini-batches and performs RDD (Resilient Distributed Datasets) transformations on those mini-batches of data.

MLlib (Machine Learning Library)

MLlib 是一个分布式机器学习框架，位于 Spark 之上，因为其分布式基于内存的 Spark 架构。根据 MLlib 开发人员对交替最小二乘法 (ALS) 实现进行的基准测试，它比 Hadoop 基于磁盘的版本快九倍。 Apache Mahout （在 Mahout 获得 Spark 接口之前）。

MLlib is a distributed machine learning framework above Spark because of the distributed memory-based Spark architecture. It is, according to benchmarks, done by the MLlib developers against the Alternating Least Squares (ALS) implementations. Spark MLlib is nine times as fast as the Hadoop disk-based version of Apache Mahout (before Mahout gained a Spark interface).

GraphX

GraphX 是一个分布式图处理框架，位于 Spark 之上。它提供了一个用于表示图形计算的 API，该 API 可以使用 Pregel 抽象 API 对用户定义的图形进行建模。它还为这种抽象提供了优化的运行时。

GraphX is a distributed graph-processing framework on top of Spark. It provides an API for expressing graph computation that can model the user-defined graphs by using Pregel abstraction API. It also provides an optimized runtime for this abstraction.