Logstash 简明教程

Logstash - Supported Inputs

Logstash 支持来自不同来源的大量日志。它与下面说明的著名来源协同工作。

Logstash supports a huge range of logs from different sources. It is working with famous sources as explained below.

Collect Logs from Metrics

系统事件和其他时间活动记录在指标中。Logstash 可以从系统指标中访问日志，并使用过滤器对其进行处理。这有助于以自定义方式向用户显示事件的动态信息。指标会根据指标过滤器的 flush_interval setting 刷新，默认情况下，它设置为 5 秒。

System events and other time activities are recorded in metrics. Logstash can access the log from system metrics and process them using filters. This helps to show the user the live feed of the events in a customized manner. Metrics are flushed according to the flush_interval setting of metrics filter and by default; it is set to 5 seconds.

我们通过收集和分析运行 Logstash 的事件并显示命令提示符上的动态信息来追踪 Logstash 生成的测试指标。

We are tracking the test metrics generated by Logstash, by gathering and analyzing the events running through Logstash and showing the live feed on the command prompt.

logstash.conf

此配置包含 Logstash 提供的生成器插件，用于测试指标，并设置类型为“generated”以进行解析。在过滤阶段中，我们仅使用“if”语句处理具有生成类型行的语句。然后，指标插件计算米制设置中指定字段。指标插件会在 flush_interval 指定的每 5 秒内刷新计数。

This configuration contains a generator plugin, which is offered by Logstash for test metrics and set the type setting to “generated” for parsing. In the filtering phase, we are only processing the lines with a generated type by using the ‘if’ statement. Then, the metrics plugin counts the field specified in meter settings. The metrics plugin flushes the count after every 5 seconds specified in the flush_interval.

最后，使用格式化 codec plugin 将过滤器事件输出到标准输出（例如命令提示符）。编解码器插件使用 [events][rate_1m] 值在 1 分钟滑动窗口中输出每秒事件。

Lastly, output the filter events to a standard output like command prompt using the codec plugin for formatting. The Codec plugin is using [events][rate_1m] value to output the per second events in a 1-minute sliding window.

input {
   generator {
     	type => "generated"
   }
}
filter {
   if [type] == "generated" {
      metrics {
         meter => "events"
         add_tag => "metric"
      }
   }
}
output {
   # only emit events with the 'metric' tag
   if "metric" in [tags] {
      stdout {
         codec => line { format => "rate: %{[events][rate_1m]}"
      }
   }
}

Run Logstash

我们可以使用以下命令运行 Logstash。

We can run Logstash by using the following command.

>logsaths –f logstash.conf

stdout (command prompt)

rate: 1308.4
rate: 1308.4
rate: 1368.654529135342
rate: 1416.4796003951449
rate: 1464.974293984808
rate: 1523.3119444107458
rate: 1564.1602979542715
rate: 1610.6496496890895
rate: 1645.2184750334154
rate: 1688.7768007612485
rate: 1714.652283095914
rate: 1752.5150680019278
rate: 1785.9432934744932
rate: 1806.912181962126
rate: 1836.0070454626025
rate: 1849.5669494173826
rate: 1871.3814756851832
rate: 1883.3443123790712
rate: 1906.4879113216743
rate: 1925.9420717997118
rate: 1934.166137658981
rate: 1954.3176526556897
rate: 1957.0107444542625

Collect Logs from the Web Server

Web 服务器生成大量有关用户访问和错误的日志。Logstash 帮助使用输入插件从不同服务器中提取日志，并将其存储到集中位置。

Web servers generate a large number of logs regarding user access and errors. Logstash helps to extract the logs from different servers using input plugins and stash them in a centralized location.

我们正在从本地 Apache Tomcat 服务器的 stderr logs 中提取数据并将其存储在 output.log 中。

We are extracting the data from the stderr logs of the local Apache Tomcat Server and stashing it in the output.log.

logstash.conf

这个 Logstash 配置文件指示 Logstash 读取 apache 错误日志并添加一个名为“apache-error”的标签。通过使用文件输出插件，我们能够简单地将其发送到 output.log。

This Logstash configuration file directs Logstash to read apache error logs and add a tag named “apache-error”. We can simply send it to the output.log using the file output plugin.

input {
   file {
      path => "C:/Program Files/Apache Software Foundation/Tomcat 7.0 /logs/*stderr*"
      type => "apache-error"
   }
}
output {
   file {
      path => "C:/tpwork/logstash/bin/log/output.log"
   }
}

Run Logstash

我们可以使用以下命令运行 Logstash。

We can run Logstash by using the following command.

>Logstash –f Logstash.conf

Sample of Input log

这是示例 stderr log ，其在 Apache Tomcat 中发生服务器事件时生成。

This is the sample stderr log, which generates when the server events occur in Apache Tomcat.

C:\Program Files\Apache Software Foundation\Tomcat 7.0\logs\ tomcat7-stderr.2016-12-25.log

Dec 25, 2016 7:05:14 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-9999"]
Dec 25, 2016 7:05:14 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["ajp-bio-8009"]
Dec 25, 2016 7:05:14 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 823 ms

output.log

{
   "path":"C:/Program Files/Apache Software Foundation/Tomcat 7.0/logs/
   tomcat7-stderr.2016-12-25.log","@timestamp":"2016-12-25T11:05:27.045Z",
   "@version":"1","host":"Dell-PC",
   "message":"Dec 25, 2016 7:05:14 PM org.apache.coyote.AbstractProtocol start\r",
   "type":"apache-error","tags":[]
}
{
   "path":"C:/Program Files/Apache Software Foundation/Tomcat 7.0/logs/
   tomcat7-stderr.2016-12-25.log","@timestamp":"2016-12-25T11:05:27.045Z",
   "@version":"1","host":"Dell-PC",
   "message":"INFO: Starting ProtocolHandler [
      \"ajp-bio-8009\"]\r","type":"apache-error","tags":[]
}
{
   "path":"C:/Program Files/Apache Software Foundation/Tomcat 7.0/logs/
   tomcat7-stderr.2016-12-25.log","@timestamp":"2016-12-25T11:05:27.045Z",
   "@version":"1","host":"Dell-PC",
   "message":"Dec 25, 2016 7:05:14 PM org.apache.catalina.startup.Catalina start\r",
   "type":"apache-error","tags":[]
}
{
   "path":"C:/Program Files/Apache Software Foundation/Tomcat 7.0/logs/
   tomcat7-stderr.2016-12-25.log","@timestamp":"2016-12-25T11:05:27.045Z",
   "@version":"1","host":"Dell-PC",
   "message":"INFO: Server startup in 823 ms\r","type":"apache-error","tags":[]
}

Collect Logs from Data sources

首先，让我们了解如何配置 MySQL 以进行日志记录。在 [mysqld] 下的 MySQL 数据库服务器的 my.ini file 中添加以下行。

To start with, let us understand how to Configure MySQL for logging. Add the following lines in my.ini file of the MySQL database server under [mysqld].

在 Windows 中，它保存在 MySQL 的安装目录内，位于：

In windows, it is present inside the installation directory of MySQL, which is in −

C:\wamp\bin\mysql\mysql5.7.11

在 UNIX 中，你可以在此处找到它：– /etc/mysql/my.cnf

In UNIX, you can find it in – /etc/mysql/my.cnf

general_log_file   = "C:/wamp/logs/queries.log"
general_log = 1

logstash.conf

在这个配置文件中，文件插件用于读取 MySQL 日志并将其写入 ouput.log。

In this config file, file plugin is used to read the MySQL log and write it to the ouput.log.

input {
   file {
      path => "C:/wamp/logs/queries.log"
   }
}
output {
   file {
      path => "C:/tpwork/logstash/bin/log/output.log"
   }
}

queries.log

这是 MySQL 数据库中执行查询所生成的日志。

This is the log generated by queries executed in the MySQL database.

2016-12-25T13:05:36.854619Z   2 Query		select * from test1_users
2016-12-25T13:05:51.822475Z    2 Query	select count(*) from users
2016-12-25T13:05:59.998942Z    2 Query         select count(*) from test1_users

output.log

{
   "path":"C:/wamp/logs/queries.log","@timestamp":"2016-12-25T13:05:37.905Z",
   "@version":"1","host":"Dell-PC",
   "message":"2016-12-25T13:05:36.854619Z    2 Query\tselect * from test1_users",
   "tags":[]
}
{
   "path":"C:/wamp/logs/queries.log","@timestamp":"2016-12-25T13:05:51.938Z",
   "@version":"1","host":"Dell-PC",
   "message":"2016-12-25T13:05:51.822475Z    2 Query\tselect count(*) from users",
   "tags":[]
}
{
   "path":"C:/wamp/logs/queries.log","@timestamp":"2016-12-25T13:06:00.950Z",
   "@version":"1","host":"Dell-PC",
   "message":"2016-12-25T13:05:59.998942Z    2 Query\tselect count(*) from test1_users",
   "tags":[]
}