Linux Admin 简明教程

Linux Admin - Log Management

Systemd 改变了 CentOS Linux 管理系统日志记录的方法。系统中的每个守护进程不再将日志放置在单独位置,而是使用 tial 或 grep 等工具作为分类日志条目和过滤日志条目的主要方式, journald 为系统日志分析带来了一个单一的管理点。

Systemd has changed the way system logging is managed for CentOS Linux. Instead of every daemon on the system placing logs into individual locations than using tools such as tail or grep as the primary way of sorting and filtering log entries, journald has brought a single point of administration to analyzing system logs.

systemd 日志记录背后的主要组件包括:journal、journalctl 和 journald.conf

The main components behind systemd logging are: journal, jounralctl, and journald.conf

journald 是主要日志守护进程,调整 journald.conf 进行配置,而 journalctl 用于分析 journald 记录的事件。

journald is the main logging daemon and is configured by editing journald.conf while journalctl is used to analyze events logged by journald.

journald 记录的事件包括:内核事件、用户进程和守护进程服务。

Events logged by journald include − kernel events, user processes, and daemon services.

Set the Correct System Time Zone

在使用 journalctl 之前,我们需要确保系统时间设置正确。要做到这一点,我们需要使用 timedatectl。

Before using journalctl, we need to make sure our system time is set to the correct time. To do this, we want to use timedatectl.

让我们检查当前系统时间。

Let’s check the current system time.

[root@centos rdc]# timedatectl status
Local time: Mon 2017-03-20 00:14:49 MDT
Universal time: Mon 2017-03-20 06:14:49 UTC
RTC time: Mon 2017-03-20 06:14:49
Time zone: America/Denver (MDT, -0600)
NTP enabled: yes
NTP synchronized: yes
RTC in local TZ: no
DST active: yes
Last DST change: DST began at
              Sun 2017-03-12 01:59:59 MST
              Sun 2017-03-12 03:00:00 MDT
Next DST change: DST ends (the clock jumps one hour backwards) at
              Sun 2017-11-05 01:59:59 MDT
              Sun 2017-11-05 01:00:00 MST

[root@centos rdc]#

当前,系统对应于本地时区。如果您的系统不对应时区,那么让我们设置正确的时区。在更改设置之后,CentOS 会自动计算当前时区的时间偏移量,立即调整系统时钟。

Currently, the system is correct to the local time zone. If your system is not, let’s set the correct time zone. After changing the settings, CentOS will automatically calculate the time zone offset from the current time zone, adjusting the system clock right away.

让我们使用 timedatectl 列出所有时区:-

Let’s list all the time zones with timedatectl −

[root@centos rdc]# timedatectl list-timezones
Africa/Abidjan
Africa/Accra
Africa/Addis_Ababa
Africa/Algiers
Africa/Asmara
Africa/Bamako
Africa/Bangui
Africa/Banjul
Africa/Bissau

这是 timedatectl list-timezones 返回的输出。要找到特定本地时区,可以使用 grep 命令:

That is the contended output from timedatectl list-timezones. To find a specific local time-zone, the grep command can be used −

[root@centos rdc]# timedatectl list-timezones | grep -i "america/New_York"
America/New_York
[root@centos rdc]#

CentOS 使用的标签通常为 Country/Region,并使用下划线代替空格(New_York 与 "New York")。

The label used by CentOS is usually Country/Region with an underscore instead of space (New_York versus "New York").

现在让我们设置我们的时区:

Now let’s set our time zone −

[root@centos rdc]# timedatectl set-timezone "America/New_York"

[root@centos rdc]# date
Mon Mar 20 02:28:44 EDT 2017

[root@centos rdc]#

您的系统时钟应该自动调整时间。

Your system clock should automatically adjust the time.

Use journalctl to Analyze Logs

使用 journalctl 时的常用命令行切换:

Common command line switches when using journalctl −

Switch

Action

-k

Lists only kernel messages

-u

Lists by specific unit (httpd, sshd, etc…​)

-b

Boots the label offset

-o

Logs the output format

-p

Filters by log type (either name or number)

-F

Fieldname or fieldnamevalue

--utc

Time in UTC offset

--since

Filter by timeframe

Examine Boot Logs

首先,我们将检查和配置 CentOS Linux 中的引导日志。您会注意到的第一件事是,默认情况下,CentOS 不会存储在重启后仍然存在的引导日志。

First, we will examine and configure the boot logs in CentOS Linux. The first thing you will notice is that CentOS, by default, doesn’t store boot logging that is persistent across reboots.

要按重启实例检查引导日志,我们可以发出以下命令:

To check boot logs per reboot instance, we can issue the following command −

[root@centos rdc]# journalctl --list-boots
-4 bca6380a31a2463aa60ba551698455b5 Sun 2017-03-19 22:01:57 MDT—Sun 2017-03-19 22:11:02 MDT
-3 3aaa9b84f9504fa1a68db5b49c0c7208 Sun 2017-03-19 22:11:09 MDT—Sun 2017-03-19 22:15:03 MDT
-2 f80b231272bf48ffb1d2ce9f758c5a5f Sun 2017-03-19 22:15:11 MDT—Sun 2017-03-19 22:54:06 MDT
-1 a071c1eed09d4582a870c13be5984ed6 Sun 2017-03-19 22:54:26 MDT—Mon 2017-03-20 00:48:29 MDT
 0 9b4e6cdb43b14a328b1fa6448bb72a56 Mon 2017-03-20 00:48:38 MDT—Mon 2017-03-20 01:07:36 MDT

[root@centos rdc]#

在重启系统之后,我们可以看到另一个条目。

After rebooting the system, we can see another entry.

[root@centos rdc]# journalctl --list-boots
-5 bca6380a31a2463aa60ba551698455b5 Sun 2017-03-19 22:01:57 MDT—Sun 2017-03-19 22:11:02 MDT
-4 3aaa9b84f9504fa1a68db5b49c0c7208 Sun 2017-03-19 22:11:09 MDT—Sun 2017-03-19 22:15:03 MDT
-3 f80b231272bf48ffb1d2ce9f758c5a5f Sun 2017-03-19 22:15:11 MDT—Sun 2017-03-19 22:54:06 MDT
-2 a071c1eed09d4582a870c13be5984ed6 Sun 2017-03-19 22:54:26 MDT—Mon 2017-03-20 00:48:29 MDT
-1 9b4e6cdb43b14a328b1fa6448bb72a56 Mon 2017-03-20 00:48:38 MDT—Mon 2017-03-20 01:09:57 MDT
 0 aa6aaf0f0f0d4fcf924e17849593d972 Mon 2017-03-20 01:10:07 MDT—Mon 2017-03-20 01:12:44 MDT

[root@centos rdc]#

现在,让我们检查最后的引导日志实例:

Now, let’s examine the last boot logging instance −

root@centos rdc]# journalctl -b -5
-- Logs begin at Sun 2017-03-19 22:01:57 MDT, end at Mon 2017-03-20 01:20:27 MDT. --
Mar 19 22:01:57 localhost.localdomain systemd-journal[97]: Runtime journal is using 8.0M
(max allowed 108.4M
Mar 19 22:01:57 localhost.localdomain kernel: Initializing cgroup subsys cpuset
Mar 19 22:01:57 localhost.localdomain kernel: Initializing cgroup subsys cpu
Mar 19 22:01:57 localhost.localdomain kernel: Initializing cgroup subsys cpuacct
Mar 19 22:01:57 localhost.localdomain kernel: Linux version 3.10.0514.6.2.el7.x86_64
(builder@kbuilder.dev.
Mar 19 22:01:57 localhost.localdomain kernel: Command line:
BOOT_IMAGE=/vmlinuz-3.10.0-514.6.2.el7.x86_64 ro
Mar 19 22:01:57 localhost.localdomain kernel: Disabled fast string operations
Mar 19 22:01:57 localhost.localdomain kernel: e820: BIOS-provided physical RAM map:

以下是我们上次启动的简洁输出。我们还可以参考数小时、数天、数周、数月乃至数年的启动日志。但是,CentOS 默认情况下不存储永久启动日志。要启用永久存储启动日志,我们需要进行一些配置更改 −

Above is the condensed output from our last boot. We could also refer back to a boot log from hours, days, weeks, months, and even years. However, by default CentOS doesn’t store persistent boot logs. To enable persistently storing boot logs, we need to make a few configuration changes −

  1. Make central storage points for boot logs

  2. Give proper permissions to a new log folder

  3. Configure journald.conf for persistent logging

Configure Boot Location for Persistent Boot Logs

journald 将要存储持久化启动日志的初始位置是 /var/log/journal。由于默认情况下不存在,因此让我们创建它 −

The initial place journald will want to store persistent boot logs is /var/log/journal. Since this doesn’t exist by default, let’s create it −

[root@centos rdc]# mkdir /var/log/journal

现在,让我们给目录适当的权限 journald 守护程序访问权限 −

Now, let’s give the directory proper permissions journald daemon access −

systemd-tmpfiles --create --prefix /var/log/journal

最后,让我们告诉 journald 它应该存储永久启动日志。在 vim 或您喜欢的文本编辑器中,打开 /etc/systemd/jounrald.con”。

Finally, let’s tell journald it should store persistent boot logs. In vim or your favorite text editor, open /etc/systemd/jounrald.conf".

# See journald.conf(5) for details.
[Journal]=Storage=peristent

我们关注的行是 Storage=。首先删除注释 #,然后更改为上面所示的 Storage = persistent 。保存并重启您的 CentOS 系统,并注意在运行 journalctl list-boots 时应该有多个条目。

The line we are concerned with is, Storage=. First remove the comment #, then change to Storage = persistent as depicted above. Save and reboot your CentOS system and take care that there should be multiple entries when running journalctl list-boots.

Note -来自 VPS 提供商的不断变化的机器 ID 可能会导致 journald 无法存储持久启动日志。对于这种情况,有很多变通办法。最好浏览发布在 CentOS 管理员论坛上的当前修复程序,而不是遵循已找到合理 VPS 变通办法的那些人的可信建议。

Note − A constantly changing machine-id like that from a VPS provider can cause journald to fail at storing persistent boot logs. There are many workarounds for such a scenario. It is best to peruse the current fixes posted to CentOS Admin forums, than follow the trusted advice from those who have found plausible VPS workarounds.

要检查特定启动日志,我们只需使用 journald --list-boots,使用 -b 开关进行偏移。因此,为了检查第二个启动日志,我们将使用 −

To examine a specific boot log, we simply need to get each offset using journald --list-boots the offset with the -b switch. So to check the second boot log we’d use −

journalctl -b -2

如果没有指定启动日志偏移量,-b 的默认值将始终是最新的启动日志。

The default for -b with no boot log offset specified will always be the current boot log after the last reboot.

Analyze Logs by Log Type

来自 journald 的事件被编号并归类为 7 种不同类型 −

Events from journald are numbered and categorized into 7 separate types −

0 - emerg   :: System is unusable
1 - alert   :: Action must be taken immediatly
2 - crit    :: Action is advised to be taken immediatly
3 - err     :: Error effecting functionality of application
4 - warning :: Usually means a common issue that can affect security or usilbity
5 - info    :: logged informtation for common operations
6 - debug   :: usually disabled by default to troubleshoot functionality

因此,如果我们想查看所有警告,可以通过 journalctl 发布以下命令 −

Hence, if we want to see all warnings the following command can be issued via journalctl −

[root@centos rdc]# journalctl -p 4
-- Logs begin at Sun 2017-03-19 22:01:57 MDT, end at Wed 2017-03-22 22:33:42 MDT. --
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: RSDP 00000000000f6a10 00024
(v02 PTLTD )
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: XSDT 0000000095eea65b 0005C
(v01 INTEL  440BX    06040000 VMW  01
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: FACP 0000000095efee73 000F4
(v04 INTEL  440BX    06040000 PTL  00
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: DSDT 0000000095eec749 1272A
(v01 PTLTD  Custom   06040000 MSFT 03
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: FACS 0000000095efffc0 00040
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: BOOT 0000000095eec721 00028
(v01 PTLTD  $SBFTBL$ 06040000  LTP 00
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: APIC 0000000095eeb8bd 00742
(v01 PTLTD  ? APIC   06040000  LTP 00
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: MCFG 0000000095eeb881 0003C
(v01 PTLTD  $PCITBL$ 06040000  LTP 00
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: SRAT 0000000095eea757 008A8
(v02 VMWARE MEMPLUG  06040000 VMW  00
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: HPET 0000000095eea71f 00038
(v01 VMWARE VMW HPET 06040000 VMW  00
Mar 19 22:01:57 localhost.localdomain kernel: ACPI: WAET 0000000095eea6f7 00028
(v01 VMWARE VMW WAET 06040000 VMW  00
Mar 19 22:01:57 localhost.localdomain kernel: Zone ranges:
Mar 19 22:01:57 localhost.localdomain kernel:   DMA      [mem 0x000010000x00ffffff]
Mar 19 22:01:57 localhost.localdomain kernel:   DMA32    [mem 0x010000000xffffffff]
Mar 19 22:01:57 localhost.localdomain kernel:   Normal   empty
Mar 19 22:01:57 localhost.localdomain kernel: Movable zone start for each node
Mar 19 22:01:57 localhost.localdomain kernel: Early memory node ranges
Mar 19 22:01:57 localhost.localdomain kernel:   node   0: [mem 0x000010000x0009dfff]
Mar 19 22:01:57 localhost.localdomain kernel:   node   0: [mem 0x001000000x95edffff]
Mar 19 22:01:57 localhost.localdomain kernel:   node   0: [mem 0x95f000000x95ffffff]
Mar 19 22:01:57 localhost.localdomain kernel: Built 1 zonelists in Node order,
mobility grouping on.  Total pages: 60
Mar 19 22:01:57 localhost.localdomain kernel: Policy zone: DMA32
Mar 19 22:01:57 localhost.localdomain kernel: ENERGY_PERF_BIAS: Set to
'normal', was 'performance'

以上显示了系统过去 4 天内所有警告。

The above shows all warnings for the past 4 days on the system.

以 systemd 查看和浏览日志的新方式确实需要一点点实践和研究才能熟悉。然而,由于不同的输出格式和特别注意使所有打包的守护程序日志通用,因此值得采用。journald 在传统日志分析方法上提供了极大的灵活性和效率。

The new way of viewing and perusing logs with systemd does take little practice and research to become familiar with. However, with different output formats and particular notice to making all packaged daemon logs universal, it is worth embracing. journald offers great flexibility and efficiency over traditional log analysis methods.