新建一臺虛擬機elk4部署logstash

[root@elk4 ~]# yum install -y jdk-11.0.15_linux-x64_bin.rpm[root@elk4 ~]# yum install -y logstash-7.6.1.rpm

命令方式

[root@elk4 bin]# /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'

elasticsearch輸出插件

[root@elk4 conf.d]# pwd[root@elk4 conf.d]# vim test.confinput {stdin { }
}output {stdout {}elasticsearch {hosts => "192.168.92.31:9200"index => "logstash-%{+YYYY.MM.dd}"}
}

[root@elk4 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf

啟動成功后錄入數據，ctrl+c退出

elasticsearch-head插件

安裝依賴

[root@k8s1 ~]# yum install -y bzip2[root@k8s1 ~]# tar jxf phantomjs-2.1.1-linux-x86_64.tar.bz2[root@k8s1 ~]# cd phantomjs-2.1.1-linux-x86_64[root@k8s1 phantomjs-2.1.1-linux-x86_64]# cp bin/phantomjs /usr/local/bin/[root@k8s1 ~]# yum install -y fontconfig[root@k8s1 ~]# phantomjs

安裝插件

[root@k8s1 ~]# rpm -ivh nodejs-9.11.2-1nodesource.x86_64.rpm[root@k8s1 ~]# yum install -y unzip[root@k8s1 ~]# unzip elasticsearch-head-master.zip[root@k8s1 ~]# cd elasticsearch-head-master/[root@k8s1 elasticsearch-head-master]# npm install  --registry=https://registry.npm.taobao.org

[root@k8s1 elasticsearch-head-master]# vim _site/app.js

啟動服務

[root@k8s1 elasticsearch-head-master]# npm run start &

[root@k8s1 elasticsearch-head-master]# netstat -antlp|grep :9100

修改es配置

[root@elk1 ~]# vim /etc/elasticsearch/elasticsearch.ymlhttp.cors.enabled: true
http.cors.allow-origin: "*"[root@elk1 ~]# systemctl  restart elasticsearch.service

訪問：192.168.92.11:9100

file輸入插件

[root@elk4 conf.d]# vim es.confinput {#file {#       path => "/var/log/messages"#       start_position => "beginning"#  }syslog {}}output {stdout {}elasticsearch {hosts => "192.168.92.31:9200"index => "rsyslog-%{+YYYY.MM.dd}"}
}

[root@elk4 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf

.sincedb文件保存文件讀取進度，避免數據冗余讀取

[root@elk4 file]# pwd[root@elk4 file]# l.

sincedb文件一共6個字段

1. inode編號
2. 文件系統的主要設備號
3. 文件系統的次要設備號
4. 文件中的當前字節偏移量
5. 最后一個活動時間戳（浮點數）
6. 與此記錄匹配的最后一個已知路徑

刪除后重新讀取

[root@elk4 file]# rm -f .sincedb_452905a167cf4509fd08acb964fdb20c

syslog?插件

logstash偽裝成日志服務器

[root@elk4 conf.d]# vim test.confinput {syslog {}
}output {stdout {}elasticsearch {hosts => "192.168.92.31:9200"index => "syslog-%{+YYYY.MM.dd}"}}

[root@elk4 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf

配置客戶端日志輸出

[root@server1?~]#?vim?/etc/rsyslog.conf

去掉以下行的注釋

[root@elk1 ~]# systemctl  restart rsyslog.service

多行過濾插件

從server1拷貝模板文件

[root@elk1 elasticsearch]# pwd[root@elk1 elasticsearch]# scp my-es.log elk4:/var/log/

[root@elk4 conf.d]# vim my-es-log.confinput {file {path => "/var/log/my-es.log"start_position => "beginning"codec => multiline {pattern => "^\["negate => truewhat => previous}}}output {stdout {}elasticsearch {hosts => "192.168.92.31:9200"index => "myeslog-%{+YYYY.MM.dd}"}}

[root@elk4 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf

grok過濾

[root@elk4 ~]# yum install -y httpd[root@elk4 ~]# systemctl  enablel --now httpd[root@elk4 ~]# echo www.westos.org > /var/www/html/index.html

訪問此站點生成日志信息

ab -c 1 -n 500 http://192.168.92.34/index.html

編寫文件

[root@elk4 conf.d]# vim grok.confinput {file {path => "/var/log/httpd/access_log"start_position => "beginning"}
}filter {grok {match => { "message" => "%{HTTPD_COMBINEDLOG}" }}
}output {stdout {}elasticsearch {hosts => "192.168.92.31:9200"index => "apachelog-%{+YYYY.MM.dd}"}}

[root@elk4 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/grok.conf