我正在尝试解析我的Nginx日志并将其发送到ElasticSearch进行分析。这是我的logstash配置:
logstash.conf
input {
file {
path => "/var/log/nginx/access.log"
type => "nginx_access"
}
}
filter {
if [type] == "nginx_access" {
grok {
patterns_dir => "/home/daspiyush0/logstash-6.1.2/patterns"
match => { "message" => "%{NGINX_ACCESS}" }
remove_tag => ["nginx_access", "_grokparsefailure"]
add_field => {
"type" => "nginx_access"
}
remove_field => ["program"]
}
date {
match => ["time_local", "dd/MMM/YYYY:HH:mm:ss Z"]
target => "@timestamp"
remove_field => "time_local"
}
useragent {
source => "user_agent"
target => "useragent"
remove_field => "user_agent"
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => true
template_overwrite => true
template => "/home/daspiyush0/logstash-6.1.2/templates/es_template.json"
index => "logstash-%{+YYYY.MM.dd}"
}
}
/home/daspiyush0/logstash-6.1.2/patterns/nginx_access
METHOD (OPTIONS|GET|HEAD|POST|PUT|DELETE|TRACE|CONNECT)
NGINX_ACCESS %{IPORHOST:visitor_ip} - - \[%{HTTPDATE:time_local}\] "%
{METHOD:method} %{URIPATHPARAM:path} HTTP/%{NUMBER:http_version}" %
{INT:status} %{INT:body_bytes_sent} "%{URI:referer}" "%
{QS:user_agent}"
Nginx日志样本
127.0.0.1 - - [19/Jan/2018:12:03:52 +0530] "GET /favicon.ico HTTP/1.1"
502 575 "http://127.0.0.1/" "Mozilla/5.0 (X11; Linux x86_64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94
Safari/537.36" "-"
形成样本文件
{
"_index": "logstash-2018.01.19",
"_type": "nginx_access",
"_id": "AWENLcPHlWpuWFLYWlZ6",
"_score": 1,
"_source": {
"@version": "1",
"tags": [
"_grokparsefailure"
],
"host": "daspiyush0-thinkpad-e450",
"type": "nginx_access",
"path": "/var/log/nginx/access.log",
"@timestamp": "2018-01-19T06:49:17.684Z",
"message": "127.0.0.1 - - [19/Jan/2018:12:19:17 +0530] \"GET / HTTP/1.1\" 502 575 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36\" \"-\""
}
}
使用上述过滤器配置时,我无法正常解析。我究竟做错了什么?
最佳答案
您的模式几乎是正确的。问题是它的最新标记"%
{QS:user_agent}"
。 QS
代表quoted string
,但是您用另一双双引号将QS字段括起来。如下更改模式,它应该可以工作:
NGINX_ACCESS %{IPORHOST:visitor_ip} - - \[%{HTTPDATE:time_local}\] "%{METHOD:method} %{URIPATHPARAM:path} HTTP/%{NUMBER:http_version}" %{INT:status} %{INT:body_bytes_sent} "%{URI:referer}" %{QS:user_agent}
关于elasticsearch - 用Logstash进行Grok解析失败,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/48335776/