csv - 将慢日志发送到 .csv 文件？

Question

我正在使用logstash 2.4.0和logstash 2.4.0 我想使用logstash 将慢日志发送到.csv 输出文件。我的配置文件是这样的

      input {
  file {
    path => "D:\logstash-2.4.0\logstash-2.4.0\bin\rachu.log"
    start_position => "beginning"
  }
}

filter {
   grok {
        match => [ "message", 

"\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]\[%{LOGLEVEL:LEVEL}%{SPACE}\]\[%{DATA:QUERY}\]%{SPACE}\[%{DATA:QUERY1}\]%{SPACE}\[%{DATA:INDEX-NAME}\]\[%{DATA:SHARD}\]%{SPACE}took\[%{DATA:TOOK}\],%{SPACE}took_millis\[%{DATA:TOOKM}\], types\[%{DATA:types}\], stats\[%{DATA:stats}\],search_type\[%{DATA:search_type}\], total_shards\[%{NUMBER:total_shards}\], source\[%{DATA:source_query}\], extra_source\[%{DATA:extra_source}\],"]
   }
}
output {
   csv {
      fields => ["TIMESTAMP","LEVEL","QUERY","QUERY1","INDEX-NAME","SHARD","TOOK","TOOKM","types","stats","search_type","total_shards","source_query","extra_source"]
      path => "D:\logstash-2.4.0\logstash-2.4.0\bin\logoutput.csv"
      spreadsheet_safe => false
   }

}

Answer 1

csv 过滤器在您的上下文中没有用处。它的目标是解析传入的 CSV 数据，但这不是您所拥有的。您需要首先使用 grok 过滤器解析日志行，然后才能将其正确发送到 csv 输出:

filter {
   grok {
      match => {"message" => "\[%{TIMESTAMP_ISO8601:TIMESTAMP}\]\[%{LOGLEVEL:LOGLEVEL} \]\[%{DATA:QUERY}\] \[%{WORD:QUERY1}\] \[%{WORD:INDEX}\]\[%{INT:SHARD}\] took\[%{BASE10NUM:TOOK}ms\], took_millis\[%{BASE10NUM:took_millis}\], types\[%{DATA:types}\], stats\[%{DATA:stats}\], search_type\[%{DATA:search_type}\], total_shards\[%{INT:total_shards}\], source\[%{DATA:source}\], extra_source\[%{DATA:extra_source}\]"}
   }
}
output {
   csv {
      fields => ["TIMESTAMP","LOGLEVEL","QUERY","QUERY1","INDEX-NAME","SHARD","TOOK","took_millis","types","stats","search_type","total_shards","source_query","extra_source"]
      path => "F:\logstash-5.1.1\logstash-5.1.1\finaloutput1"
      spreadsheet_safe => false
   }
}

注意:这在 Logstash 5.1.1 上还不起作用，因为 this open issue 。它应该很快就会得到修复，但与此同时，这可以在 Logstash 2.4 上运行。