Kibana 简明教程
Kibana - Loading Sample Data
我们已经了解如何将数据从 logstash 上传到 elasticsearch。我们将在这里使用 logstash 和 elasticsearch 上传数据。但是对于我们需要的带有日期、经度和纬度字段的数据，我们将在后面的章节中了解。如果我们没有 CSV 文件，我们还将看到如何在 Kibana 中直接上传数据。
We have seen how to upload data from logstash to elasticsearch. We will upload data using logstash and elasticsearch here. But about the data that has date, longitude and latitudefields which we need to use, we will learn in the upcoming chapters. We will also see how to upload data directly in Kibana, if we do not have a CSV file.
在本章中，我们将介绍以下主题 −
In this chapter, we will cover following topics −
-
Using Logstash upload data having date, longitude and latitude fields in Elasticsearch
-
Using Dev tools to upload bulk data
Using Logstash upload for data having fields in Elasticsearch
我们将使用 CSV 格式的数据，并且是从 Kaggle.com 获取的数据，其中包含可以用于分析的数据。
We are going to use data in the form of CSV format and the same is taken from Kaggle.com which deals with data that you can use for an analysis.
此处使用的家庭医疗服务数据是从网站 Kaggle.com 获取的。
The data home medical visits to be used here is picked up from site Kaggle.com.
以下是 CSV 文件中可用的字段 −
The following are the fields available for the CSV file −
["Visit_Status","Time_Delay","City","City_id","Patient_Age","Zipcode","Latitude","Longitude",
"Pathology","Visiting_Date","Id_type","Id_personal","Number_Home_Visits","Is_Patient_Minor","Geo_point"]Home_visits.csv 如下所示 −
The Home_visits.csv is as follows −
以下是要与 logstash 一起使用的配置文件 −
The following is the conf file to be used with logstash −
input {
file {
path => "C:/kibanaproject/home_visits.csv"
start_position => "beginning"
sincedb_path => "NUL"
}
}
filter {
csv {
separator => ","
columns =>
["Visit_Status","Time_Delay","City","City_id","Patient_Age",
"Zipcode","Latitude","Longitude","Pathology","Visiting_Date",
"Id_type","Id_personal","Number_Home_Visits","Is_Patient_Minor","Geo_point"]
}
date {
match => ["Visiting_Date","dd-MM-YYYY HH:mm"]
target => "Visiting_Date"
}
mutate {convert => ["Number_Home_Visits", "integer"]}
mutate {convert => ["City_id", "integer"]}
mutate {convert => ["Id_personal", "integer"]}
mutate {convert => ["Id_type", "integer"]}
mutate {convert => ["Zipcode", "integer"]}
mutate {convert => ["Patient_Age", "integer"]}
mutate {
convert => { "Longitude" => "float" }
convert => { "Latitude" => "float" }
}
mutate {
rename => {
"Longitude" => "[location][lon]"
"Latitude" => "[location][lat]"
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "medicalvisits-%{+dd.MM.YYYY}"
}
stdout {codec => json_lines }
}默认情况下，logstash 将要在 elasticsearch 中上传的所有内容都视为字符串。在您的 CSV 文件中，如果有日期字段，您需要执行以下操作以获取日期格式。
By default, logstash considers everything to be uploaded in elasticsearch as string. Incase your CSV file has date field you need to do following to get the date format.
For date field −
For date field −
date {
match => ["Visiting_Date","dd-MM-YYYY HH:mm"]
target => "Visiting_Date"
}对于地理位置，elasticsearch 的理解如下 −
In-case of geo location, elasticsearch understands the same as −
"location": {
"lat":41.565505000000044,
"lon": 2.2349995750000695
}因此，我们需要确保经度和纬度采用 elasticsearch 所需的格式。因此，我们首先需要将经度和纬度转换为浮点数，然后再重命名它，使其作为 location JSON 对象的一部分，其中包含 lat 和 lon 。这里显示了代码 −
So we need to make sure we have Longitude and Latitude in the format elasticsearch needs it. So first we need to convert longitude and latitude to float and later rename it so that it is available as part of location json object with lat and lon. The code for the same is shown here −
mutate {
convert => { "Longitude" => "float" }
convert => { "Latitude" => "float" }
}
mutate {
rename => {
"Longitude" => "[location][lon]"
"Latitude" => "[location][lat]"
}
}如要将字段转换成整数，请使用以下代码 −
For converting fields to integers, use the following code −
mutate {convert => ["Number_Home_Visits", "integer"]}
mutate {convert => ["City_id", "integer"]}
mutate {convert => ["Id_personal", "integer"]}
mutate {convert => ["Id_type", "integer"]}
mutate {convert => ["Zipcode", "integer"]}
mutate {convert => ["Patient_Age", "integer"]}处理完字段后，运行以下命令将数据上传到 elasticsearch −
Once the fields are taken care, run the following command to upload the data in elasticsearch −
-
Go inside Logstash bin directory and run the following command.
logstash -f logstash_homevisists.conf-
Once done you should see the index mentioned in logstash conf file in elasticsearch as shown below −
现在，我们可以在上述上载的索引上创建索引模式，并进一步使用它来创建可视化效果。
We can now create index pattern on above index uploaded and use it further for creating visualization.
Using Dev Tools to Upload Bulk Data
我们将从 Kibana UI 使用 Dev Tools。Dev Tools 有助于将数据上载到 Elasticsearch，而无需使用 Logstash。我们可以发布、放置、删除、搜索使用 Dev Tools 在 Kibana 中想要的数据。
We are going to use Dev Tools from Kibana UI. Dev Tools is helpful to upload data in Elasticsearch, without using Logstash. We can post, put, delete, search the data we want in Kibana using Dev Tools.
在本节中，我们将尝试将示例数据加载到 Kibana 本身。我们可以使用它来练习示例数据并在 Kibana 功能中进行操作以很好地理解 Kibana。
In this section, we will try to load sample data in Kibana itself. We can use it to practice with the sample data and play around with Kibana features to get a good understanding of Kibana.
让我们从以下 URL 获取 json 数据并在 Kibana 中上载相同的数据。同样，您可以尝试将任何示例 json 数据加载到 Kibana 内。
Let us take the json data from the following url and upload the same in Kibana. Similarly, you can try any sample json data to be loaded inside Kibana.
在我们开始上载示例数据之前，我们需要让 json 数据带有在 Elasticsearch 中使用的索引。当我们使用 logstash 上载它时，logstash 会负责添加索引，用户不必关心 Elasticsearch 所需的索引。
Before we start to upload the sample data, we need to have the json data with indices to be used in elasticsearch. When we upload it using logstash, logstash takes care to add the indices and the user does not have to bother about the indices which are required by elasticsearch.
Normal Json Data
[
{"type":"act","line_id":1,"play_name":"Henry IV",
"speech_number":"","line_number":"","speaker":"","text_entry":"ACT I"},
{"type":"scene","line_id":2,"play_name":"Henry IV",
"speech_number":"","line_number":"","speaker":"","text_entry":"SCENE I.London. The palace."},
{"type":"line","line_id":3,"play_name":"Henry IV",
"speech_number":"","line_number":"","speaker":"","text_entry":
"Enter KING HENRY, LORD JOHN OF LANCASTER, the
EARL of WESTMORELAND, SIR WALTER BLUNT, and others"}
]与 Kibana 一起使用的 json 代码必须带有索引，如下所示：
The json code to used with Kibana has to be with indexed as follows −
{"index":{"_index":"shakespeare","_id":0}}
{"type":"act","line_id":1,"play_name":"Henry IV",
"speech_number":"","line_number":"","speaker":"","text_entry":"ACT I"}
{"index":{"_index":"shakespeare","_id":1}}
{"type":"scene","line_id":2,"play_name":"Henry IV",
"speech_number":"","line_number":"","speaker":"",
"text_entry":"SCENE I. London. The palace."}
{"index":{"_index":"shakespeare","_id":2}}
{"type":"line","line_id":3,"play_name":"Henry IV",
"speech_number":"","line_number":"","speaker":"","text_entry":
"Enter KING HENRY, LORD JOHN OF LANCASTER, the EARL
of WESTMORELAND, SIR WALTER BLUNT, and others"}请注意，json 文件中还有附加数据 − {"index":{"_index":"nameofindex","_id":key}} 。
Note that there is an additional data that goes in the jsonfile −{"index":{"_index":"nameofindex","_id":key}}.
为了让任何示例 json 文件与 Elasticsearch 兼容，我们这里有一个用 php 编写的小代码，它将输出 json 文件，该 json 文件给出 Elasticsearch 想要格式。
To convert any sample json file compatible with elasticsearch, here we have a small code in php which will output the json file given to the format which elasticsearch wants −
PHP Code
<?php
$myfile = fopen("todo.json", "r") or die("Unable to open file!"); // your json
file here
$alldata = fread($myfile,filesize("todo.json"));
fclose($myfile);
$farray = json_decode($alldata);
$afinalarray = [];
$index_name = "todo";
$i=0;
$myfile1 = fopen("todonewfile.json", "w") or die("Unable to open file!"); //
writes a new file to be used in kibana dev tool
foreach ($farray as $a => $value) {
$_index = json_decode('{"index": {"_index": "'.$index_name.'", "_id": "'.$i.'"}}');
fwrite($myfile1, json_encode($_index));
fwrite($myfile1, "\n");
fwrite($myfile1, json_encode($value));
fwrite($myfile1, "\n");
$i++;
}
?>我们从 https://jsonplaceholder.typicode.com/todos 获取 todo json 文件，并使用 php 代码将其转换为我们需要在 Kibana 中上载的格式。
We have taken the todo json file from https://jsonplaceholder.typicode.com/todos and use php code to convert to the format we need to upload in Kibana.
要加载示例数据，请打开 dev tools 标签，如下所示：
To load the sample data, open the dev tools tab as shown below −
现在，我们将使用如上所示的控制台。我们将获取通过 php 代码运行后得到的 json 数据。
We are now going to use the console as shown above. We will take the json data which we got after running it through php code.
在 dev tools 中用于上载 json 数据的命令为：
The command to be used in dev tools to upload the json data is −
POST _bulk请注意，我们创建的索引名称为 todo。
Note that the name of the index we are creating is todo.
单击绿色按钮后，数据将被上载，您可以按照以下步骤检查索引是否在 Elasticsearch 中创建：
Once you click the green button the data is uploaded, you can check if the index is created or not in elasticsearch as follows −
您可以在 dev tools 本身中检查相同的内容，如下所示：
You can check the same in dev tools itself as follows −
Command −
Command −
GET /_cat/indices
如果您想在您的索引：todo 中搜索一些内容，您可以按照以下步骤操作：
If you want to search something in your index:todo , you can do that as shown below −
Command in dev tool
Command in dev tool
GET /todo/_search
显示的上述搜索结果如下 −
The output of the above search is as shown below −
它显示了 todoindex 中存在的所有记录。我们获得的总记录数是 200。
It gives all the records present in the todoindex. The total records we are getting is 200.
