elasticsearch - 无法使用Logstash(7.6.2)和xml过滤器插件设置解析和设置时间戳

标签 elasticsearch logstash

环境详细信息:
ELK堆栈7.6.2
Windows 10

通过Logstash进行索引时，无法从日志中替换/设置 flex 搜索时间戳。它只是添加为新字段，而不替换原始字段。它只是添加了“_dateparsefailure”标签而没有任何其他信息。

我怀疑日期过滤器不起作用。

我的样本日志数据:

<log4j:event logger="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" timestamp="1530819710045" level="WARN" thread="27"><log4j:message>registrarCheck.bookingWizardController.TryUpdatebookingCareOptions(): bookingCareOptionId: CenterBasedCare, bookingId: 5745493, bookingregistrarsCount: 5, IsEditbooking: False, IsEditbookingStep2Modified: False, IsMemberShip: False</log4j:message><log4j:properties><log4j:data name="log4japp" value="/LM/W3SVC/2/ROOT-1-131752914805620482" /><log4j:data name="log4net:Identity" value="user1" /><log4j:data name="log4jmachinename" value="webserver1" /><log4j:data name="log4net:UserName" value="SomeOrganization\!svc-app-identity" /><log4j:data name="log4net:HostName" value="webserver1" /></log4j:properties><log4j:locationInfo class="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" method="Warn" file="c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.ApplicationBlocks\Logging\Logging.cs" line="283" /></log4j:event>
<log4j:event logger="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" timestamp="1530819760731" level="ERROR" thread="15"><log4j:message>ERROR from EasyDraft API for funding accountid-&gt;0-&gt;Name: firstname lastname-&gt;Card number is invalid</log4j:message><log4j:properties><log4j:data name="log4japp" value="/LM/W3SVC/2/ROOT-1-131752914805620482" /><log4j:data name="log4net:Identity" value="user1" /><log4j:data name="log4jmachinename" value="webserver1" /><log4j:data name="log4net:UserName" value="SomeOrganization\!svc-app-identity" /><log4j:data name="log4net:HostName" value="webserver1" /></log4j:properties><log4j:locationInfo class="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" method="Error" file="c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.ApplicationBlocks\Logging\Logging.cs" line="139" /></log4j:event>
<log4j:event logger="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" timestamp="1530819760856" level="ERROR" thread="15"><log4j:message>Error in controller: effective username: user1, identity username: user1, machine name: webserver1 
Client Name: [zzz Test ESomeApplication], Contract Id: [7ee17d62-d292-e511-b173-005056991898]
, Person Id: [143658262]
, Client ID: [b33442b3-d192-e511-b173-005056991898], Contract Relationship ID: [4529625]
, Person Fullname: [firstname lastname].
, Full Name: [firstname lastname], CRM ID: [a64c97b1-8a80-e811-b738-005056991899]</log4j:message><log4j:properties><log4j:data name="log4japp" value="/LM/W3SVC/2/ROOT-1-131752914805620482" /><log4j:data name="log4net:Identity" value="user1" /><log4j:data name="log4jmachinename" value="webserver1" /><log4j:data name="log4net:UserName" value="SomeOrganization\!svc-app-identity" /><log4j:data name="log4net:HostName" value="webserver1" /></log4j:properties><log4j:throwable>SomeOrganization.SomeApplication.BusinessLogic.Security.SomeOrganizationSomeApplicationException: Exception of type 'SomeOrganization.SomeApplication.BusinessLogic.Security.SomeOrganizationSomeApplicationException' was thrown.
   at SomeOrganization.SomeApplication.BusinessLogic.PaymentAccount.Save() in c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\SomeApplication\SomeOrganization.SomeApplication.BusinessLogic\PaymentAccount.cs:line 415
   at Csla.BusinessBase`1.Save(Boolean forceUpdate) in C:\andre\mainline\SomeApplicationNewDevelopment\CSLA\Source-4.3.12\Csla\BusinessBase.cs:line 163
   at Csla.BusinessBase`1.Csla.Core.ISavable.Save(Boolean forceUpdate) in C:\andre\mainline\SomeApplicationNewDevelopment\CSLA\Source-4.3.12\Csla\BusinessBase.cs:line 350
   at SomeOrganization.Shared.Web.ApplicationBlocks.Controllers.CustomCslaMvcController.SaveObject[T](T item, Action`1 updateModel, Boolean forceUpdate) in c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.Web.ApplicationBlocks\Controllers\CustomCslaMvcController.cs:line 171</log4j:throwable><log4j:locationInfo class="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" method="Error" file="c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.ApplicationBlocks\Logging\Logging.cs" line="165" /></log4j:event>
<log4j:event logger="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" timestamp="1530824089499" level="ERROR" thread="41"><log4j:message>Error Occured while Save Login in Class Login &amp; Method : Save For Username : tegh14</log4j:message><log4j:properties><log4j:data name="log4japp" value="/LM/W3SVC/2/ROOT-2-131752976869399121" /><log4j:data name="log4net:UserName" value="SomeOrganization\!svc-app-identity" /><log4j:data name="log4jmachinename" value="webserver1" /><log4j:data name="log4net:HostName" value="webserver1" /></log4j:properties><log4j:throwable>System.Security.Authentication.AuthenticationException: We can�t find that username and/or password.  If you are trying to register for the first time using your employer�s credentials, select the Create Your Profile link below. If you are having trouble accessing the site, feel free to call us at none-one-CARES in the United States or Canada, 0800 000 000 in the United Kingdom, or 0800 000 000 in Ireland.
   at SomeOrganization.Shared.BusinessLogic.Security.Login.Save() in c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.BusinessLogic\Security\Login.cs:line 547</log4j:throwable><log4j:locationInfo class="SomeOrganization.Shared.ApplicationBlocks.Logging.Logger" method="Error" file="c:\Builds\5\mainline\Main.SomeApplication\Sources\mainline\Main\Shared\SomeOrganization.Shared.ApplicationBlocks\Logging\Logging.cs" line="165" /></log4j:event>
<log4j:event logger="SomeOrganisation.Shared.ApplicationBlocks.Logging.Logger" timestamp="1587880949425" level="WARN" thread="47"><log4j:message>User mphilpunla-&gt;LoginWithSAML-&gt;lobuniqueId 19153694</log4j:message><log4j:properties><log4j:data name="log4jmachinename" value="webserver2" /><log4j:data name="log4japp" value="/LM/W3SVC/2/ROOT-1-132323544167926323" /><log4j:data name="log4net:UserName" value="SomeOrganisation\!svc-lob-apps" /><log4j:data name="log4net:Identity" value="" /><log4j:data name="log4net:HostName" value="webserver2" /></log4j:properties><log4j:locationInfo class="SomeOrganisation.Shared.ApplicationBlocks.Logging.Logger" method="Warn" file="E:\TFS2018agent\agent\_work\96\s\Shared\SomeOrganisation.Shared.ApplicationBlocks\Logging\Logging.cs" line="294" /></log4j:event>

我的logstash配置文件:
input { 
    file {
      path => ["C:/Users/maskedUsername/Desktop/stackoverflow-log4net.txt"]
      start_position => "beginning"
      file_sort_by => "last_modified"
      file_sort_direction => "desc"
      sincedb_path => "NUL"
      type => "appl"
      codec => multiline {
          pattern => "^<log4j:event"
          negate => true
          what => "previous"
      }
    }
 }

filter {
  if [type] == "appl" {
    grok {
        add_tag => [ "groked" ]
        match => ["message", ".*"]
        remove_tag => ["_grokparsefailure"]
    }
    xml {
      source => "message"
      remove_namespaces => true
      target => "log4jevent"
      xpath => [ "//event/@timestamp", "timestamp" ]
      xpath => [ "//event/@level", "loglevel" ]
      xpath => [ "/event/message/text()", "message" ]
      xpath => [ "/event/throwable/text()", "exception" ]
      xpath => [ "//event/properties/data[@name='log4jmachinename']/@value", "machinename" ]
      xpath => [ "//event/properties/data[@name='log4japp']/@value", "app" ]
      xpath => [ "//event/properties/data[@name='log4net:UserName']/@value", "username" ]
      xpath => [ "//event/properties/data[@name='log4net:Identity']/@value", "identity" ]
      xpath => [ "//event/properties/data[@name='log4net:HostName']/@value", "hostname" ]
    }
    mutate {
      remove_field => ["type", "tags", "message"]
    }
    date {
        match => [ "timestamp","UNIX" ]
        target => "@timestamp"
        remove_field => ["timestamp"]
    }
  }
}

output {
  elasticsearch { 
  hosts => ["localhost:9200"] 
  index => "log4jevents"
  document_type => "log4jevent"
  }
  stdout { codec => rubydebug }
}

我的 flex 搜索文档数据:
{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 4,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "log4jevents",
        "_type" : "log4jevent",
        "_id" : "kACTLnIBpHd52XYqFAtE",
        "_score" : 1.0,
        "_source" : {
          "timestamp" : [
            "1530819710045"
          ],
          "tags" : [
            "_dateparsefailure"
          ],
          "host" : "localhost",
          "machinename" : [
            "webserver1"
          ],
          "identity" : [
            "user1"
          ],
          "username" : [
            "SomeOrganization\\!svc-app-identity"
          ],
          "@timestamp" : "2020-05-19T20:14:49.672Z",
          "@version" : "1",
          "hostname" : [
            "webserver1"
          ],
          "loglevel" : [
            "WARN"
          ],
          "app" : [
            "/LM/W3SVC/2/ROOT-1-131752914805620482"
          ],
          "path" : "C:/Users/MaskedUsername/Desktop/stackoverflow-log4net.txt"
        }
      },
      {
        "_index" : "log4jevents",
        "_type" : "log4jevent",
        "_id" : "jwCTLnIBpHd52XYqFAtE",
        "_score" : 1.0,
        "_source" : {
          "timestamp" : [
            "1530819760731"
          ],
          "tags" : [
            "_dateparsefailure"
          ],
          "host" : "localhost",
          "machinename" : [
            "webserver1"
          ],
          "identity" : [
            "user1"
          ],
          "username" : [
            "SomeOrganization\\!svc-app-identity"
          ],
          "@timestamp" : "2020-05-19T20:14:49.700Z",
          "@version" : "1",
          "hostname" : [
            "webserver1"
          ],
          "loglevel" : [
            "ERROR"
          ],
          "app" : [
            "/LM/W3SVC/2/ROOT-1-131752914805620482"
          ],
          "path" : "C:/Users/MaskedUsername/Desktop/stackoverflow-log4net.txt"
        }
      },
      {
        "_index" : "log4jevents",
        "_type" : "log4jevent",
        "_id" : "kQCTLnIBpHd52XYqFAtE",
        "_score" : 1.0,
        "_source" : {
          "timestamp" : [
            "1530824089499"
          ],
          "tags" : [
            "_dateparsefailure"
          ],
          "host" : "localhost",
          "machinename" : [
            "webserver1"
          ],
          "username" : [
            "SomeOrganization\\!svc-app-identity"
          ],
          "@timestamp" : "2020-05-19T20:14:49.738Z",
          "@version" : "1",
          "hostname" : [
            "webserver1"
          ],
          "loglevel" : [
            "ERROR"
          ],
          "app" : [
            "/LM/W3SVC/2/ROOT-2-131752976869399121"
          ],
          "path" : "C:/Users/MaskedUsername/Desktop/stackoverflow-log4net.txt",
          "exception" : [
            "System.Security.Authentication.AuthenticationException: We can�t find that username and/or password.  If you are trying to register for the first time using your employer�s credentials, select the Create Your Profile link below. If you are having trouble accessing the site, feel free to call us at none-one-CARES in the United States or Canada, 0800 000 000 in the United Kingdom, or 0800 000 000 in Ireland.\n   at SomeOrganization.Shared.BusinessLogic.Security.Login.Save() in c:\\Builds\\5\\mainline\\Main.SomeApplication\\Sources\\mainline\\Main\\Shared\\SomeOrganization.Shared.BusinessLogic\\Security\\Login.cs:line 547"
          ]
        }
      },
      {
        "_index" : "log4jevents",
        "_type" : "log4jevent",
        "_id" : "kgCTLnIBpHd52XYqFAvT",
        "_score" : 1.0,
        "_source" : {
          "timestamp" : [
            "1530819760856"
          ],
          "tags" : [
            "_dateparsefailure"
          ],
          "host" : "localhost",
          "machinename" : [
            "webserver1"
          ],
          "identity" : [
            "user1"
          ],
          "username" : [
            "SomeOrganization\\!svc-app-identity"
          ],
          "@timestamp" : "2020-05-19T20:14:49.732Z",
          "@version" : "1",
          "hostname" : [
            "webserver1"
          ],
          "loglevel" : [
            "ERROR"
          ],
          "app" : [
            "/LM/W3SVC/2/ROOT-1-131752914805620482"
          ],
          "path" : "C:/Users/MaskedUsername/Desktop/stackoverflow-log4net.txt",
          "exception" : [
            "SomeOrganization.SomeApplication.BusinessLogic.Security.SomeOrganizationSomeApplicationException: Exception of type 'SomeOrganization.SomeApplication.BusinessLogic.Security.SomeOrganizationSomeApplicationException' was thrown.\n   at SomeOrganization.SomeApplication.BusinessLogic.PaymentAccount.Save() in c:\\Builds\\5\\mainline\\Main.SomeApplication\\Sources\\mainline\\Main\\SomeApplication\\SomeOrganization.SomeApplication.BusinessLogic\\PaymentAccount.cs:line 415\n   at Csla.BusinessBase`1.Save(Boolean forceUpdate) in C:\\andre\\mainline\\SomeApplicationNewDevelopment\\CSLA\\Source-4.3.12\\Csla\\BusinessBase.cs:line 163\n   at Csla.BusinessBase`1.Csla.Core.ISavable.Save(Boolean forceUpdate) in C:\\andre\\mainline\\SomeApplicationNewDevelopment\\CSLA\\Source-4.3.12\\Csla\\BusinessBase.cs:line 350\n   at SomeOrganization.Shared.Web.ApplicationBlocks.Controllers.CustomCslaMvcController.SaveObject[T](T item, Action`1 updateModel, Boolean forceUpdate) in c:\\Builds\\5\\mainline\\Main.SomeApplication\\Sources\\mainline\\Main\\Shared\\SomeOrganization.Shared.Web.ApplicationBlocks\\Controllers\\CustomCslaMvcController.cs:line 171"
          ]
        }
      }
    ]
  }
}

我在这里想念什么？

最佳答案

UNIX模式期望您的时间是自纪元以来的秒unit时间，是10位数字，但是您的timestamp字段是自纪元以来的unit时间(以毫秒为单位)，是13位数字。

您应该改用UNIX_MS模式。

date {
    match => [ "timestamp","UNIX_MS" ]
    target => "@timestamp"
    remove_field => ["timestamp"]
}

编辑:

如果timestamp字段是整数，则上述过滤器可以正常工作，但是即使您只有一个信息，xml过滤器似乎也将数据存储在数组中，因此在这种情况下，timestamp字段位于索引0中，然后是该字段在过滤器中需要是[timestamp][0]
date {
    match => [ "[timestamp][0]","UNIX_MS" ]
    target => "@timestamp"
    remove_field => ["timestamp"]
}

用以下消息模拟。
{ "msg": "sample message", "timestamp": ["1530819710045"] }

输出为:
{
    "@timestamp" => 2018-07-05T19:41:50.045Z,
          "host" => "elk",
      "@version" => "1",
           "msg" => "sample message"
}

关于elasticsearch - 无法使用Logstash(7.6.2)和xml过滤器插件设置解析和设置时间戳，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/61900140/

上一篇：powershell - 如何在Powershell中使用cd命令

下一篇：regex - 正则表达式可以在所有.net引擎中在线工作，而不能在Powershell中工作？

相关文章：

docker - 如何使用 docker 将 metricbeat 连接到 elasticsearch 和 kibana

elasticsearch - 使用Logstash解析百万个日志

elasticsearch - 如何通过AWS VPC使用Elastic Cloud的Logstash从MySQL(AWS RDS)导入数据？

elasticsearch - logstash kv 过滤器，使用动态映射将字符串转换为整数

redis - 分布式logstash + redis + elasticsearch设置

mongodb - Elasticsearch mongo-connector KeyError _id

elasticsearch - 安装Kopf-Elasticsearch

.net - 使用NEST在Elasticsearch中检索未知数量的文档

elasticsearch - 在 Elasticsearch 中过滤没有分数的查询

elasticsearch - (logstash) 仅索引来自 kafka 输入的 elasticsearch 中的特定数据

©2024 IT工具网  联系我们