regex - 自定义GROK过滤器-Logstash-> Elasticsearch

Question

我有一个被捕获并发送到logstash的日志，该日志的格式为

22304999    5   400.OUTPUT_SERVICE.510  submit  The limit has been exceeded. Please use a different option. 2.54.44.221 /api/output/v3/contract/:PCID/order /api/output/v3/contract/:pcid/order https://www.example.org/output/ PUT 400 2017-09-28T15:50:57.843176Z

我正在尝试创建一个自定义grok过滤器，以在将其发送到elasticsearch之前添加 header 字段。

我的目标是这样的

 SessionID   => "22304999"
 HitNumber   => "5"
 FactValue   => "400.OUTPUT_SERVICE.510"
 DimValue1   => "submit"
 ErrMessage  => "The limit has been exceeded. Please use a different option."
 IP          => "2.54.44.221"
 TLT_URL     => "/api/output/v3/contract/:PCID/order"
 URL         => "/api/output/v3/contract/:pcid/order"
 Refferer    => "https://www.example.org/output/"
 Method      => "PUT"
 StatsCode   => "400"
 ReqTime     => "2017-09-28T15:50:57.843176Z"

我对此并不陌生，因此只能尝试了解如何应用和测试它，例如，我将从一个空的过滤器开始，

filter {
   grok {
     match => { "message" => "" }
   }
 }

我的第一个问题是match => { "message" => "" }是消息只是一条日志行吗？什么定义“消息”？

我的日志和我想要的字段之间用制表符分隔，每个制表符后都有一个新字段，这会使我试图实现的目标变得更容易，而不是寻找模式，我可以只寻找下一个制表符吗？

失败的话，有人可以为我的一个 Realm 提供一个例子，我应该可以完成其余的工作。

Answer 1

正则表达式:(?<SessionID>\S+)\s+(?<HitNumber>\S+)\s+(?<FactValue>\S+)\s+(?<DimValue1>\S+)\s+(?<ErrMessage>.+)\s+(?<IP>(?:\d{1,3}\.){3}\d{1,3})\s+(?<TLT_URL>\S+)\s+(?<URL>\S+)\s+(?<Refferer>\S+)\s+(?<Method>\S+)\s+(?<StatsCode>\S+)\s+(?<ReqTime>\S+)
详细信息:

(?<>)命名为捕获组

\S与任何非空白字符

匹配

\d匹配数字，{n,m}匹配n和m乘以

+一次和无限次匹配

Regex demo

输出:

{
  "SessionID": [
    [
      "22304999"
    ]
  ],
  "HitNumber": [
    [
      "5"
    ]
  ],
  "FactValue": [
    [
      "400.OUTPUT_SERVICE.510"
    ]
  ],
  "DimValue1": [
    [
      "submit"
    ]
  ],
  "ErrMessage": [
    [
      "The limit has been exceeded. Please use a different option."
    ]
  ],
  "IP": [
    [
      "2.54.44.221"
    ]
  ],
  "TLT_URL": [
    [
      "/api/output/v3/contract/:PCID/order"
    ]
  ],
  "URL": [
    [
      "/api/output/v3/contract/:pcid/order"
    ]
  ],
  "Refferer": [
    [
      "https://www.example.org/output/"
    ]
  ],
  "Method": [
    [
      "PUT"
    ]
  ],
  "StatsCode": [
    [
      "400"
    ]
  ],
  "ReqTime": [
    [
      "2017-09-28T15:50:57.843176Z"
    ]
  ]
}