Fluentd

syslog に出力されたメッセージを必要に応じてメール送信したい
td-agent ではなく、Ruby Gem から Fluentd をインストール
Fluentd をサービス化する
基本的なインストール手順は Install by Ruby Gem - Fluentd に記載

root@ubuntu:~# ulimit -n
1024

# /etc/security/limits.conf
#
#Each line describes a limit for a user in the form:
#
#<domain>        <type>  <item>  <value>
#
#Where:
#<domain> can be:
#        - a user name
#        - a group name, with @group syntax
#        - the wildcard *, for default entry
#        - the wildcard %, can be also used with %group syntax,
#                 for maxlogin limit
#        - NOTE: group and wildcard limits are not applied to root.
#          To apply a limit to the root user, <domain> must be
#          the literal username root.
#
#<type> can have the two values:
#        - "soft" for enforcing the soft limits
#        - "hard" for enforcing hard limits
#
#<item> can be one of the following:
#        - core - limits the core file size (KB)
#        - data - max data size (KB)
#        - fsize - maximum filesize (KB)
#        - memlock - max locked-in-memory address space (KB)
#        - nofile - max number of open file descriptors
#        - rss - max resident set size (KB)
#        - stack - max stack size (KB)
#        - cpu - max CPU time (MIN)
#        - nproc - max number of processes
#        - as - address space limit (KB)
#        - maxlogins - max number of logins for this user
#        - maxsyslogins - max number of logins on the system
#        - priority - the priority to run user process with
#        - locks - max number of file locks the user can hold
#        - sigpending - max number of pending signals
#        - msgqueue - max memory used by POSIX message queues (bytes)
#        - nice - max nice priority allowed to raise to values: [-20, 19]
#        - rtprio - max realtime priority
#        - chroot - change root to directory (Debian-specific)
#
#<domain>      <type>  <item>         <value>
#

#*               soft    core            0
#root            hard    core            100000
#*               hard    rss             10000
#@student        hard    nproc           20
#@faculty        soft    nproc           20
#@faculty        hard    nproc           50
#ftp             hard    nproc           0
#ftp             -       chroot          /ftp
#@student        -       maxlogins       4

# End of file

# End of file の前に追加
vi /etc/security/limits.conf

#<domain>      <type>  <item>         <value>
 root          soft    nofile         65536
 root          hard    nofile         65536
 *             soft    nofile         65536
 *             hard    nofile         65536

reboot

root@ubuntu:~# ulimit -n
65536

root@ubuntu:~# apt update
Hit:1 http://jp.archive.ubuntu.com/ubuntu disco InRelease
Get:2 http://jp.archive.ubuntu.com/ubuntu disco-updates InRelease [97.5 kB]
Get:3 http://jp.archive.ubuntu.com/ubuntu disco-backports InRelease [88.8 kB]
Get:4 http://jp.archive.ubuntu.com/ubuntu disco-security InRelease [97.5 kB]
Get:5 http://jp.archive.ubuntu.com/ubuntu disco-updates/main amd64 Packages [301 kB]

～～～ 途中省略 ～～～

Get:14 http://jp.archive.ubuntu.com/ubuntu disco-security/main Translation-en [80.3 kB]
Get:15 http://jp.archive.ubuntu.com/ubuntu disco-security/main amd64 c-n-f Metadata [4,984 B]
Get:16 http://jp.archive.ubuntu.com/ubuntu disco-security/universe amd64 Packages [261 kB]
Get:17 http://jp.archive.ubuntu.com/ubuntu disco-security/universe Translation-en [77.1 kB]
Get:18 http://jp.archive.ubuntu.com/ubuntu disco-security/universe amd64 c-n-f Metadata [6,512 B]
Fetched 1,798 kB in 1s (1,776 kB/s)
Reading package lists... Done

root@ubuntu:~# apt-get install -y ruby ruby-dev gcc make
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  binutils binutils-common binutils-x86-64-linux-gnu cpp cpp-8 fonts-lato gcc-8 gcc-8-base javascript-common libasan5
  libatomic1 libbinutils libc-dev-bin libc6-dev libcc1-0 libgcc-8-dev libgmp-dev libgmpxx4ldbl libgomp1 libisl19 libitm1
  libjs-jquery liblsan0 libmpc3 libmpx2 libquadmath0 libruby2.5 libtsan0 libubsan1 linux-libc-dev manpages-dev rake
  ruby-did-you-mean ruby-minitest ruby-net-telnet ruby-power-assert ruby-test-unit ruby-xmlrpc ruby2.5 ruby2.5-dev ruby2.5-doc
  rubygems-integration unzip zip
Suggested packages:
  binutils-doc cpp-doc gcc-8-locales gcc-multilib autoconf automake libtool flex bison gdb gcc-doc gcc-8-multilib gcc-8-doc
  libgcc1-dbg libgomp1-dbg libitm1-dbg libatomic1-dbg libasan5-dbg liblsan0-dbg libtsan0-dbg libubsan1-dbg libmpx2-dbg
  libquadmath0-dbg apache2 | lighttpd | httpd glibc-doc gmp-doc libgmp10-doc libmpfr-dev make-doc ri bundler
The following NEW packages will be installed:
  binutils binutils-common binutils-x86-64-linux-gnu cpp cpp-8 fonts-lato gcc gcc-8 gcc-8-base javascript-common libasan5
  libatomic1 libbinutils libc-dev-bin libc6-dev libcc1-0 libgcc-8-dev libgmp-dev libgmpxx4ldbl libgomp1 libisl19 libitm1
  libjs-jquery liblsan0 libmpc3 libmpx2 libquadmath0 libruby2.5 libtsan0 libubsan1 linux-libc-dev make manpages-dev rake ruby
  ruby-dev ruby-did-you-mean ruby-minitest ruby-net-telnet ruby-power-assert ruby-test-unit ruby-xmlrpc ruby2.5 ruby2.5-dev
  ruby2.5-doc rubygems-integration unzip zip
0 upgraded, 48 newly installed, 0 to remove and 13 not upgraded.
Need to get 40.5 MB of archives.
After this operation, 172 MB of additional disk space will be used.
Get:1 http://jp.archive.ubuntu.com/ubuntu disco/main amd64 fonts-lato all 2.0-2 [2,698 kB]
Get:2 http://jp.archive.ubuntu.com/ubuntu disco/main amd64 binutils-common amd64 2.32-7ubuntu4 [200 kB]
Get:3 http://jp.archive.ubuntu.com/ubuntu disco/main amd64 libbinutils amd64 2.32-7ubuntu4 [468 kB]
Get:4 http://jp.archive.ubuntu.com/ubuntu disco/main amd64 binutils-x86-64-linux-gnu amd64 2.32-7ubuntu4 [1,852 kB]
Get:5 http://jp.archive.ubuntu.com/ubuntu disco/main amd64 binutils amd64 2.32-7ubuntu4 [3,384 B]

～～～ 途中省略 ～～～

Fetched 40.5 MB in 1s (50.4 MB/s)
Extracting templates from packages: 100%
Selecting previously unselected package fonts-lato.
(Reading database ... 67344 files and directories currently installed.)
Preparing to unpack .../00-fonts-lato_2.0-2_all.deb ...
Unpacking fonts-lato (2.0-2) ...
Selecting previously unselected package binutils-common:amd64.
Preparing to unpack .../01-binutils-common_2.32-7ubuntu4_amd64.deb ...
Unpacking binutils-common:amd64 (2.32-7ubuntu4) ...

～～～ 途中省略 ～～～

Setting up javascript-common (11) ...
Setting up manpages-dev (4.16-1) ...
Setting up fonts-lato (2.0-2) ...
Setting up ruby-power-assert (1.1.1-1) ...
Setting up unzip (6.0-22ubuntu1) ...

～～～ 途中省略 ～～～

Setting up ruby2.5-dev:amd64 (2.5.5-1ubuntu1) ...
Setting up ruby2.5 (2.5.5-1ubuntu1) ...
Setting up ruby-dev:amd64 (1:2.5.1) ...
Setting up ruby (1:2.5.1) ...
Setting up rake (12.3.1-3) ...
Processing triggers for mime-support (3.60ubuntu1) ...
Processing triggers for libc-bin (2.29-0ubuntu2) ...
Processing triggers for man-db (2.8.5-2) ...

root@ubuntu:~# export GEM_HOME=${HOME}/.gem/ruby
root@ubuntu:~# export PATH=${HOME}/.gem/ruby/bin:$PATH
root@ubuntu:~# gem install fluentd -v 1.7.3
Building native extensions. This could take a while...
Successfully installed msgpack-1.3.1
Fetching: yajl-ruby-1.4.1.gem (100%)
Building native extensions. This could take a while...
Successfully installed yajl-ruby-1.4.1

～～～ 途中省略 ～～～

Installing ri documentation for strptime-0.2.3
Parsing documentation for dig_rb-1.0.1
Installing ri documentation for dig_rb-1.0.1
Parsing documentation for fluentd-1.7.3
Installing ri documentation for fluentd-1.7.3
Done installing documentation for msgpack, yajl-ruby, cool.io, sigdump, serverengine, http_parser.rb, concurrent-ruby, tzinfo, tzinfo-data, strptime, dig_rb, fluentd after 13 seconds
12 gems installed

root@ubuntu:~# gem list fluentd

*** LOCAL GEMS ***

fluentd (1.7.3)

root@ubuntu:~# fluentd --setup ./fluent
Installed ./fluent/fluent.conf.

root@ubuntu:~# fluentd -c ./fluent/fluent.conf -vv&
root@ubuntu:~# echo '{"json":"message"}' | fluent-cat debug.test
root@ubuntu:~# pkill -f fluentd
connect failed: Connection refused - connect(2) for "127.0.0.1" port 24224
2019-10-21 17:52:34 +0900 [info]: fluent/log.rb:322:info: parsing config file is succeeded path="./fluent/fluent.conf"
2019-10-21 17:52:34 +0900 [trace]: fluent/log.rb:281:trace: registered output plugin 'stdout'
2019-10-21 17:52:34 +0900 [trace]: fluent/log.rb:281:trace: registered buffer plugin 'memory'
2019-10-21 17:52:34 +0900 [trace]: fluent/log.rb:281:trace: registered formatter plugin 'stdout'
2019-10-21 17:52:34 +0900 [trace]: fluent/log.rb:281:trace: registered formatter plugin 'json'

～～～ 途中省略 ～～～

2019-10-21 17:52:34 +0900 [info]: fluent/log.rb:322:info: using configuration file: <ROOT>
  <source>
    @type forward
    @id forward_input
  </source>
  <source>
    @type http
    @id http_input
    port 8888
  </source>
  <source>
    @type monitor_agent
    @id monitor_agent_input
    port 24220
  </source>
  <source>
    @type debug_agent
    @id debug_agent_input
    bind "127.0.0.1"
    port 24230
  </source>
  <match debug.**>
    @type stdout
    @id stdout_output
  </match>
  <match system.**>
    @type forward
    @id forward_output
    <server>
      host "192.168.0.11"
    </server>
    <secondary>
      <server>
        host "192.168.0.12"
      </server>
    </secondary>
  </match>
</ROOT>
2019-10-21 17:52:34 +0900 [info]: fluent/log.rb:322:info: starting fluentd-1.7.3 pid=6745 ruby="2.5.5"
2019-10-21 17:52:34 +0900 [info]: fluent/log.rb:322:info: spawn command to main:  cmdline=["/usr/bin/ruby2.5", "-Eascii-8bit:ascii-8bit", "/root/.gem/ruby/bin/fluentd", "-c", "./fluent/fluent.conf", "-vv", "--under-supervisor"]
2019-10-21 17:52:34 +0900 [info]: fluent/log.rb:322:info: gem 'fluentd' version '1.7.3'
2019-10-21 17:52:34 +0900 [info]: fluent/log.rb:322:info: adding match pattern="debug.**" type="stdout"
2019-10-21 17:52:34 +0900 [trace]: #0 fluent/log.rb:281:trace: registered output plugin 'stdout'

～～～ 途中省略 ～～～

2019-10-21 17:52:36 +0900 [trace]: #0 [forward_input] connected fluent socket addr="127.0.0.1" port=36556
2019-10-21 17:52:36 +0900 [trace]: #0 [forward_input] accepted fluent socket addr="127.0.0.1" port=36556
2019-10-21 17:52:34.333258285 +0900 debug.test: {"json":"message"}
2019-10-21 17:52:36 +0900 [debug]: #0 fluent/log.rb:302:debug: fluentd main process get SIGTERM
2019-10-21 17:52:36 +0900 [debug]: #0 fluent/log.rb:302:debug: getting start to shutdown main process

～～～ 途中省略 ～～～

2019-10-21 17:52:40 +0900 [debug]: #0 fluent/log.rb:302:debug: calling terminate on input plugin type=:monitor_agent plugin_id="monitor_agent_input"
2019-10-21 17:52:40 +0900 [debug]: #0 fluent/log.rb:302:debug: calling terminate on input plugin type=:debug_agent plugin_id="debug_agent_input"
2019-10-21 17:52:40 +0900 [debug]: #0 fluent/log.rb:302:debug: calling terminate on output plugin type=:stdout plugin_id="stdout_output"
2019-10-21 17:52:40 +0900 [debug]: #0 fluent/log.rb:302:debug: calling terminate on output plugin type=:forward plugin_id="forward_output"
2019-10-21 17:52:40 +0900 [info]: fluent/log.rb:322:info: Worker 0 finished with status 0

root@ubuntu:~# mkdir /etc/fluentd
root@ubuntu:~# fluentd --setup /etc/fluentd
Installed /etc/fluentd/fluent.conf.
root@ubuntu:~# mv /etc/fluentd/fluent.conf /etc/fluentd/fluentd.conf
root@ubuntu:~# touch /var/log/fluentd

vi /usr/lib/systemd/system/fluentd.service

[Unit]
Description=Fluentd: Open Source Data Collector.
Documentation=https://www.fluentd.org/
After=network-online.target
Wants=network-online.target

[Service]
User=root
Group=root
LimitNOFILE=65536
Environment=GEM_HOME=/var/lib/gems/2.5.0/
Environment=GEM_PATH=/var/lib/gems/2.5.0
Environment=FLUENT_CONF=/etc/fluentd/fluentd.conf
Environment=BIN_ARGS="--log /var/log/fluentd --daemon /run/fluentd.pid"
PIDFile=/run/fluentd.pid
Type=forking
ExecStart=/usr/local/bin/fluentd $BIN_ARGS
ExecStop=/bin/kill -TERM ${MAINPID}
ExecReload=/bin/kill -HUP ${MAINPID}
Restart=always
TimeoutStopSec=120

[Install]
WantedBy=multi-user.target

パスやバージョンなどは、環境に合わせて変更のこと

root@ubuntu:~# systemctl daemon-reload
root@ubuntu:~# systemctl start fluentd
root@ubuntu:~# systemctl status fluentd
● fluentd.service - Fluentd: Open Source Data Collector.
   Loaded: loaded (/lib/systemd/system/fluentd.service; disabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-10-22 12:44:39 JST; 4s ago
     Docs: https://www.fluentd.org/
  Process: 21272 ExecStart=/usr/local/bin/fluentd $BIN_ARGS (code=exited, status=0/SUCCESS)
 Main PID: 21288 (fluentd)
    Tasks: 8 (limit: 4535)
   Memory: 45.6M
   CGroup: /system.slice/fluentd.service
           tq21288 /usr/bin/ruby2.5 /usr/local/bin/fluentd --log /var/log/fluentd --daemon /run/fluentd.pid
           mq21293 /usr/bin/ruby2.5 -Eascii-8bit:ascii-8bit /usr/local/bin/fluentd --log /var/log/fluentd --daemon /run/fluentd.

Oct 22 12:44:39 ubuntu systemd[1]: Starting Fluentd: Open Source Data Collector....
Oct 22 12:44:39 ubuntu systemd[1]: Started Fluentd: Open Source Data Collector..

root@ubuntu:~# systemctl is-enabled fluentd
disabled
root@ubuntu:~# systemctl enable fluentd
Created symlink /etc/systemd/system/multi-user.target.wants/fluentd.service → /lib/systemd/system/fluentd.service.
root@ubuntu:~# systemctl is-enabled fluentd
enabled

reboot

root@ubuntu:~# systemctl status fluentd
● fluentd.service - Fluentd: Open Source Data Collector.
   Loaded: loaded (/lib/systemd/system/fluentd.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-10-22 12:49:35 JST; 2min 14s ago
     Docs: https://www.fluentd.org/
  Process: 690 ExecStart=/usr/local/bin/fluentd $BIN_ARGS (code=exited, status=0/SUCCESS)
 Main PID: 828 (fluentd)
    Tasks: 8 (limit: 4535)
   Memory: 59.3M
   CGroup: /system.slice/fluentd.service
           tq828 /usr/bin/ruby2.5 /usr/local/bin/fluentd --log /var/log/fluentd --daemon /run/fluentd.pid
           mq833 /usr/bin/ruby2.5 -Eascii-8bit:ascii-8bit /usr/local/bin/fluentd --log /var/log/fluentd --daemon /run/fluentd.pi

Oct 22 12:49:29 ubuntu systemd[1]: Starting Fluentd: Open Source Data Collector....
Oct 22 12:49:35 ubuntu systemd[1]: Started Fluentd: Open Source Data Collector..

設定方針は logrotate の設定 を参照

vi /etc/logrotate.d/fluentd

/var/log/fluentd
{
        rotate 366
        daily
        missingok
        notifempty
        nocompress
        dateext
        dateformat _%Y%m%d-%H%M%S
        olddir /var/log/archives
        sharedscripts
        postrotate
                pid=/run/fluentd.pid
                test -s $pid && kill -USR1 "$(cat $pid)"
        endscript
}

設定は、方針や環境に合わせて変更のこと

/usr/sbin/logrotate /etc/logrotate.conf