Apache Kafka 简明教程

Apache Kafka - WorkFlow

目前，我们已讨论了 Kafka 的核心概念。现在让我们重点介绍一下 Kafka 的工作流。

As of now, we discussed the core concepts of Kafka. Let us now throw some light on the workflow of Kafka.

Kafka 仅仅是由一个或多个分区分割而成的主题集合。Kafka 分区是有序的消息序列，其中每条消息都由其索引（称为偏移量）标识。Kafka 集群中的所有数据都是分区的离散并集。传入消息被写入分区的末尾，消息被消费者依次读取。通过将消息复制到不同的代理来提供持久性。

Kafka is simply a collection of topics split into one or more partitions. A Kafka partition is a linearly ordered sequence of messages, where each message is identified by their index (called as offset). All the data in a Kafka cluster is the disjointed union of partitions. Incoming messages are written at the end of a partition and messages are sequentially read by consumers. Durability is provided by replicating messages to different brokers.

Kafka 采用快速、可靠、持久、容错和零停机方式提供发布-订阅和基于队列的消息系统。在这两种情况下，生产者只需将消息发送到主题，消费者即可根据需要选择任一类型的消息系统。让我们按照下一部分中的步骤来了解消费者如何选择其所选择的消息系统。

Kafka provides both pub-sub and queue based messaging system in a fast, reliable, persisted, fault-tolerance and zero downtime manner. In both cases, producers simply send the message to a topic and consumer can choose any one type of messaging system depending on their need. Let us follow the steps in the next section to understand how the consumer can choose the messaging system of their choice.

Workflow of Pub-Sub Messaging

以下是发布-订阅消息的逐步工作流 −

Following is the step wise workflow of the Pub-Sub Messaging −

  1. Producers send message to a topic at regular intervals.

  2. Kafka broker stores all messages in the partitions configured for that particular topic. It ensures the messages are equally shared between partitions. If the producer sends two messages and there are two partitions, Kafka will store one message in the first partition and the second message in the second partition.

  3. Consumer subscribes to a specific topic.

  4. Once the consumer subscribes to a topic, Kafka will provide the current offset of the topic to the consumer and also saves the offset in the Zookeeper ensemble.

  5. Consumer will request the Kafka in a regular interval (like 100 Ms) for new messages.

  6. Once Kafka receives the messages from producers, it forwards these messages to the consumers.

  7. Consumer will receive the message and process it.

  8. Once the messages are processed, consumer will send an acknowledgement to the Kafka broker.

  9. Once Kafka receives an acknowledgement, it changes the offset to the new value and updates it in the Zookeeper. Since offsets are maintained in the Zookeeper, the consumer can read next message correctly even during server outrages.

  10. This above flow will repeat until the consumer stops the request.

  11. Consumer has the option to rewind/skip to the desired offset of a topic at any time and read all the subsequent messages.

Workflow of Queue Messaging / Consumer Group

在队列消息系统中，一群具有相同组 ID 的消费者会订阅主题，而不是单个消费者。简单来说，订阅具有相同组 ID 的主题的消费者被视为一个组，消息在它们之间共享。让我们查看此系统的实际工作流。

In a queue messaging system instead of a single consumer, a group of consumers having the same Group ID will subscribe to a topic. In simple terms, consumers subscribing to a topic with same Group ID are considered as a single group and the messages are shared among them. Let us check the actual workflow of this system.

  1. Producers send message to a topic in a regular interval.

  2. Kafka stores all messages in the partitions configured for that particular topic similar to the earlier scenario.

  3. A single consumer subscribes to a specific topic, assume Topic-01 with Group ID as Group-1.

  4. Kafka interacts with the consumer in the same way as Pub-Sub Messaging until new consumer subscribes the same topic, Topic-01 with the same Group ID as Group-1.

  5. Once the new consumer arrives, Kafka switches its operation to share mode and shares the data between the two consumers. This sharing will go on until the number of con-sumers reach the number of partition configured for that particular topic.

  6. Once the number of consumer exceeds the number of partitions, the new consumer will not receive any further message until any one of the existing consumer unsubscribes. This scenario arises because each consumer in Kafka will be assigned a minimum of one partition and once all the partitions are assigned to the existing consumers, the new consumers will have to wait.

  7. This feature is also called as Consumer Group. In the same way, Kafka will provide the best of both the systems in a very simple and efficient manner.

Role of ZooKeeper

Apache Kafka 的一个关键依赖是 Apache Zookeeper，这是一项分布式配置与同步服务。Zookeeper 是 Kafka 代理与使用者之间的协调界面。Kafka 服务器通过 Zookeeper 丛集共享资讯。Kafka 会在 Zookeeper 中存储基本元资料，例如关于主题、代理、使用者偏移（队列阅读器）等資訊。

A critical dependency of Apache Kafka is Apache Zookeeper, which is a distributed configuration and synchronization service. Zookeeper serves as the coordination interface between the Kafka brokers and consumers. The Kafka servers share information via a Zookeeper cluster. Kafka stores basic metadata in Zookeeper such as information about topics, brokers, consumer offsets (queue readers) and so on.

由于所有关键资讯都被存储在 Zookeeper 中，而且它通常会跨其整体复制该资料，因此 Kafka 代理/Zookeeper 的故障不会影响 Kafka 丛集的状态。一旦 Zookeeper 重新启动，Kafka 将还原状态。这为 Kafka 带来了零停机时间。在领导者发生故障时，Kafka 代理之间的领导者选举也是通过使用 Zookeeper 进行的。

Since all the critical information is stored in the Zookeeper and it normally replicates this data across its ensemble, failure of Kafka broker / Zookeeper does not affect the state of the Kafka cluster. Kafka will restore the state, once the Zookeeper restarts. This gives zero downtime for Kafka. The leader election between the Kafka broker is also done by using Zookeeper in the event of leader failure.

要深入了解 Zookeeper，请参阅 zookeeper 。

To learn more on Zookeeper, please refer zookeeper

在下一章节中，让我们继续深入了解如何在您的机器上安装 Java、ZooKeeper 和 Kafka。

Let us continue further on how to install Java, ZooKeeper, and Kafka on your machine in the next chapter.