Logstash 简明教程

Logstash - Introduction

Logstash 是一种基于 filter/pipes 模式的工具，用于收集、处理和生成日志或事件。它有助于集中化实时分析来自不同来源的日志和事件。

Logstash is a tool based on the filter/pipes patterns for gathering, processing and generating the logs or events. It helps in centralizing and making real time analysis of logs and events from different sources.

Logstash 用 JRuby 编程语言编写，在 JVM 上运行，因此可以在不同平台上运行 Logstash。它从几乎所有类型的来源收集不同类型的数据，例如日志、分组、事件、事务、时间戳数据等。数据源可以是社交数据、电子商务、新闻文章、CRM、游戏数据、Web 趋势、财务数据、物联网、移动设备等。

Logstash is written on JRuby programming language that runs on the JVM, hence you can run Logstash on different platforms. It collects different types of data like Logs, Packets, Events, Transactions, Timestamp Data, etc., from almost every type of source. The data source can be Social data, E-commerce, News articles, CRM, Game data, Web trends, Financial data, Internet of Things, Mobile devices, etc.

Logstash General Features

Logstash 的一般功能如下 −

The general features of Logstash are as follows −

  1. Logstash can collect data from different sources and send to multiple destinations.

  2. Logstash can handle all types of logging data like Apache Logs, Windows Event Logs, Data over Network Protocols, Data from Standard Input and many more.

  3. Logstash can also handle http requests and response data.

  4. Logstash provides a variety of filters, which helps the user to find more meaning in the data by parsing and transforming it.

  5. Logstash can also be used for handling sensors data in internet of things.

  6. Logstash is open source and available under the Apache license version 2.0.

Logstash Key Concepts

Logstash 的关键概念如下 −

The key concepts of Logstash are as follows −

Event Object

它是 Logstash 中的主要对象，它封装了 Logstash 管道中的数据流。Logstash 使用此对象来存储输入数据和添加在过滤阶段创建的额外字段。

It is the main object in Logstash, which encapsulates the data flow in the Logstash pipeline. Logstash uses this object to store the input data and add extra fields created during the filter stage.

Logstash 为开发人员提供一个事件 API 来操作事件。在本教程中，此事件被称作各种名称，如日志数据事件、日志事件、日志数据、输入日志数据、输出日志数据等。

Logstash offers an Event API to developers to manipulate events. In this tutorial, this event is referred with various names like Logging Data Event, Log Event, Log Data, Input Log Data, Output Log Data, etc.

Pipeline

它包含 Logstash 从输入到输出的数据流阶段。输入数据输入管道，并以事件的形式进行处理。然后以用户或终端系统所需格式发送到输出目标。

It comprises of data flow stages in Logstash from input to output. The input data is entered in the pipeline and is processed in the form of an event. Then sends to an output destination in the user or end system’s desirable format.

Input

这是 Logstash 管道中的第一阶段，用于获取 Logstash 中的数据以进一步处理。Logstash 提供各种插件从不同的平台获取数据。一些最常用的插件包括文件、Syslog、Redis 和 Beats。

This is the first stage in the Logstash pipeline, which is used to get the data in Logstash for further processing. Logstash offers various plugins to get data from different platforms. Some of the most commonly used plugins are – File, Syslog, Redis and Beats.

Filter

这是 Logstash 的中间阶段，在此阶段发生实际的事件处理。开发人员可以使用 Logstash 预先定义的 Regex 模式，为事件中的字段之间的区别以及接受的输入事件的标准创建序列。

This is the middle stage of Logstash, where the actual processing of events take place. A developer can use pre-defined Regex Patterns by Logstash to create sequences for differentiating between the fields in the events and criteria for accepted input events.

Logstash 提供各种插件来帮助开发人员解析事件并将其转换成为所需的结构。一些最常用的过滤器插件包括 Grok、Mutate、Drop、Clone 和 Geoip。

Logstash offers various plugins to help the developer to parse and transform the events into a desirable structure. Some of the most commonly used filter plugins are – Grok, Mutate, Drop, Clone and Geoip.

Output

这是 Logstash 管道中的最后一个阶段，在此阶段可以将输出事件格式化为目标系统所需的结构。最后，它在完全处理后使用插件将输出事件发送到目标。一些最常用的插件包括 Elasticsearch、文件、Graphite、Statsd 等。

This is the last stage in the Logstash pipeline, where the output events can be formatted into the structure required by the destination systems. Lastly, it sends the output event after complete processing to the destination by using plugins. Some of the most commonly used plugins are – Elasticsearch, File, Graphite, Statsd, etc.

Logstash Advantages

以下几点解释了 Logstash 的各种优势。

The following points explain the various advantages of Logstash.

  1. Logstash offers regex pattern sequences to identify and parse the various fields in any input event.

  2. Logstash supports a variety of web servers and data sources for extracting logging data.

  3. Logstash provides multiple plugins to parse and transform the logging data into any user desirable format.

  4. Logstash is centralized, which makes it easy to process and collect data from different servers.

  5. Logstash supports many databases, network protocols and other services as a destination source for the logging events.

  6. Logstash uses the HTTP protocol, which enables the user to upgrade Elasticsearch versions without having to upgrade Logstash in a lock step.

Logstash Disadvantages

以下几点阐述了 Logstash 的各种劣势。

The following points explain the various disadvantages of Logstash.

  1. Logstash uses http, which negatively affects the processing of the logging data.

  2. Working with Logstash can sometimes be a little complex, as it needs a good understanding and analysis of the input logging data.

  3. Filter plugins are not generic, so, the user may need to find the correct sequence of patterns to avoid error in parsing.

在下一章中，我们将了解 ELK 堆栈是什么以及它如何帮助 Logstash。

In the next chapter, we will understand what the ELK Stack is and how it helps Logstash.