ELK 过滤插件grok对nginx日志格式化

默认nginx的访问日志是没有格式的，日志格式如下 

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

 访问日志格式如下：要想结构化处理，然后写到es里面通过kibana根据某个字段查询，统计。下面的格式也不是k/v也不是json的，所以不能使用这些插件进行处理

192.168.179.4 - - [06/Jan/2021:10:43:15 +0800] "GET /favicon.ico HTTP/1.1" 404 3650 "http://192.168.179.102/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36" "-"

这种非结构化的非常适合grok去匹配

打开kibana去测试一下grok能不能匹配到这条日志，因为有些日志里面字段是有值的，有些没有值的使用-，所以grok里面匹配使用了

%{IPV4:remote_addr} - (%{USERNAME:remote_user}|-) \[%{HTTPDATE:time_local}\] \"%{WORD:request_method} %{URIPATHPARAM:request_uri} HTTP/%{NUMBER:http_protocol}\" %{NUMBER:http_status} %{NUMBER:body_bytes_sent} \"%{GREEDYDATA:http_referer}\" \"%{GREEDYDATA:http_user_agent}\" \"(%{IPV4:http_x_forwarded_for}|-)\"

可以看出关键字段都过滤出来了，可以对相关的字段进行查询了，也可以根据该字段做可视化仪表盘 

[root@localhost ~]# cat /etc/filebeat/filebeat.yml 
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/*.log
  tags: ["nginx"]
  fields_under_root: true
  fields:
    project: microservice
    app: product

output.logstash:
  hosts: ["192.168.179.102:5044"]

[root@localhost ~]# cat /usr/local/logstash/conf.d/test.conf
input {
  beats {
  host => "0.0.0.0"
  port => 5044
 }
}

filter {
   grok {
    match => {
      "message" => "%{IPV4:remote_addr} - (%{USERNAME:remote_user}|-) \[%{HTTPDATE:time_local}\] \"%{WORD:request_method} %{URIPATHPARAM:request_uri} HTTP/%{NUMBER:http_protocol}\" %{NUMBER:http_status} %{NUMBER:body_bytes_sent} \"%{GREEDYDATA:http_referer}\" \"%{GREEDYDATA:http_user_agent}\" \"(%{IPV4:http_x_forwarded_for}|-)\""
 } 
}
  if [app] == "product" {
    mutate { 
      add_field => { 
        "[@metadata][target_index]" => "microservice-product-%{+YYYY.MM}" 
   } 
  }
} else if [app] == "gateway" {
    mutate { 
      add_field => { 
        "[@metadata][target_index]" => "microservice-gateway-%{+YYYY.MM.dd}" 
   } 
  }
} else {
    mutate { 
      add_field => { 
        "[@metadata][target_index]" => "unknown-%{+YYYY}" 
   } 
  } 
 } 
}

output {
  elasticsearch {
    hosts => "192.168.179.102:9200"
    index => "%{[@metadata][target_index]}"
 } 
}

 可以看出关键字段都过滤出来了，可以对相关的字段进行查询了，也可以根据该字段做可视化仪表盘