Logstash基础知识

Logstash简介

logstash是一个开源的数据采集工具,通过数据源采集数据.然后进行过滤,并自定义格式输出到目的地。

数据分为:

结构化数据如:mysql数据库里的表等
半结构化数据如: xml,yaml,json等
非结构化数据如:文档,图片,音频,视频等

logstash可以采集任何格式的数据,当然我们这里主要是讨论采集系统日志,服务日志等日志类型数据。

官方产品介绍:https://www.elastic.co/cn/products/logstash

input插件: 用于导入日志源

https://www.elastic.co/guide/en/logstash/current/input-plugins.html

filter插件: 用于过滤

https://www.elastic.co/guide/en/logstash/current/filter-plugins.html

output插件: 用于导出

https://www.elastic.co/guide/en/logstash/current/output-plugins.html

logstash部署

在logstash服务器上确认openjdk安装

[root@test3 ~]# java -version
openjdk version "1.8.0_352"
OpenJDK Runtime Environment (build 1.8.0_352-b08)
OpenJDK 64-Bit Server VM (build 25.352-b08, mixed mode)

在logstash服务器上安装logstash

[root@test3 ~]# wget https://artifacts.elastic.co/downloads/logstash/logstash-6.5.2.rpm
[root@test3 ~]# rpm -ivh logstash-6.5.2.rpm

配置logstash主配置文件

[root@3 ~]# cat /etc/logstash/logstash.yml |grep -v '#' |grep -v '^$'
path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d/          打开注释,并加上配置目录路径
http.host: "10.1.1.13"  打开注释,并改为本机IP(这是用于xpack监控用,但要收费,所以在这里不配置也可以)
path.logs: /var/log/logstash

启动测试

[root@test3 ~]# cd /usr/share/logstash/bin
使用下面的空输入和空输出启动测试一下
[root@test3 bin]# ./logstash -e 'input {stdin {}} output {stdout {}}'
运行后，输入字符将被stdout做为标准输出内容输出

运行后，输入字符将被stdout做为标准输出内容输出

关闭启动

测试能启动成功后,ctrl+c取消,则关闭了

另一种验证方法：

[root@test3]# vim /etc/logstash/conf.d/test.conf

input {
        stdin {
        }
}

filter {
}

output {
        stdout {
                codec => rubydebug        
        }
}


[root@test3 bin]# pwd
/usr/share/logstash/bin

[root@test3 bin]# ./logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/test.conf -r
......
Config Validation Result: OK. Exiting Logstash

--path.settings 指定logstash主配置文件目录
-f 指定片段配置文件
-t 测试配置文件是否正确
codec => rubydebug这句可写可不定,默认就是这种输出方式

[root@test3 bin]# ./logstash --path.settings /etc/logstash -r -f /etc/logstash/conf.d/test.conf
......

haha
{
    "@timestamp" => 2019-07-02T10:40:10.839Z,
       "message" => "haha",
          "host" => "vm3.cluster.com",
      "@version" => "1"
}
hehe
{
    "@timestamp" => 2019-07-02T10:40:11.794Z,
       "message" => "hehe",
          "host" => "vm3.cluster.com",
      "@version" => "1"
}

-r参数很强大,会动态装载配置文件,也就是说启动后,可以不用重启修改配置文件

日志采集

采集messages日志

这里以/var/log/messages为例,只定义input输入和output输出,不考虑过滤

[root@test3 bin]# vim /etc/logstash/conf.d/test.conf
input {
    file {
        path => "/var/log/messages"
        start_position => "beginning"
    }
}

output {
    elasticsearch{
        hosts => ["192.168.100.41:9200","192.168.100.40:9200"]
        index => "messages-%{+YYYY.MM.dd}"
    }
    stdout {
    }
}


[root@test3 bin]# ./logstash --path.settings /etc/logstash/ -r -f /etc/logstash/conf.d/test.conf  &

后台运行如果要杀掉,请使用pkill java或ps查看PID再kill -9清除

通过浏览器访问es-head验证

采集多日志源

[root@test3 bin]# vim /etc/logstash/conf.d/test.conf

input {
        file {
                path => "/var/log/messages"
                start_position => "beginning"
                type => "messages"
        }

        file {
                path => "/var/log/yum.log"
                start_position => "beginning"
                type => "yum"
        }
}

filter {

}

output {
        if [type] == "messages" {
        elasticsearch {
                hosts => ["192.168.100.41:9200","192.168.100.40:9200"]
                index => "messages-%{+YYYY-MM-dd}"
                }
        }

        if [type] == "yum" {
        elasticsearch {
                hosts => ["192.168.100.41:9200","192.168.100.40:9200"]
                index => "yum-%{+YYYY-MM-dd}"
                }
        }
}