python - 如何通过logstash将字段添加到kibana

Question

我正在使用python-logstash来写入logstash。它提供了添加其他字段的选项，但问题是所有字段都在“消息”字段下。

我必须承认这种解决方案对我不起作用:
How do I add a custom field to logstash/kibana?

我的python脚本如下所示:

LOGGER = logging.getLogger('python-logstash-logger')
LOGGER.setLevel(logging.INFO)
#LOGGER.addHandler(logstash.LogstashHandler(127.0.0.1, 5000, version=1))
LOGGER.addHandler(logstash.TCPLogstashHandler('127.0.0.1', 5000, version=1))
LOGGER.error('python-logstash: test logstash error message.')
LOGGER.info('python-logstash: test logstash info message.')
LOGGER.warning('python-logstash: test logstash warning message.')

# add extra field to logstash message
extra = {
    'test_string': 'python version: ' + repr(sys.version_info),
    'test_boolean': True,
    'test_dict': {'a': 1, 'b': 'c'},
    'test_float': 1.23,
    'test_integer': 123,
    'test_list': [1, 2, '3'],
}

LOGGER.info("python-logstash: test extra fields", extra=extra)

我的logstath confing文件是:

input {
  beats {
    port => 5044
  }
  stdin { codec => plain }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
}

我想要的只是从Extra变量中的键创建我的自定义字段，例如'test_string'。就像我说的那样，所有这些额外的变量都不会出现在“消息”字段中，而我希望该字典中的每个键都变成基巴纳语中的一个字段。如何做到这一点？

另外，我从logstash中收到以下错误(我在PowerShell中看到了它):

[ERROR][logstash.codecs.json     ][main] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unrecognized token 'mestamp': was expecting ('true', 'false' or 'null')

这可能是由于 token 损坏，如下所示:



我知道 token @version:1可能来自我的logstashHandler，但是该TIMESTAMP来自何处以及如何修复该 token ？

************************ /////// ////////// ********* *********************

我认为所有字段都落在“消息”字段中的唯一原因是该 token 已损坏。
如何修复“mestamp” token ？它来自何处？
我没有在我的python或logstash代码中设置它。

Answer 1

当我使用mutate插件时，似乎工作正常。这是我的logstash config file让我知道您是否还有问题

 input {

    http {                                                                                                        

    }   

 }

  filter {
     mutate {
        add_field => { "test_string" => "Python version 1" }
     }
   }

output {
    stdout {
  #     codec => {rubydebug}
    }   
    elasticsearch {

      hosts=> ["localhost:9200"]
      index => "so-test1"
    }   
}

这就是我在木ib中看到的

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "so-test1",
        "_type" : "_doc",
        "_id" : "XOUei28B--Dy_XuABlDq",
        "_score" : 1.0,
        "_source" : {
          "@version" : "1",
          "test_string" : "Python version 1",  **<== test string that I appended**
          "@timestamp" : "2020-01-09T16:23:17.734Z",
          "host" : "0:0:0:0:0:0:0:1",
          "message" : "hello",     **<=== message the I sent**
          "headers" : {
            "request_path" : "/",
            "postman_token" : "9e9e45a1-d6d2-445ca-9f8f-5eae9dd15320",
            "http_accept" : "*/*",
            "http_host" : "localhost:8080",
            "request_method" : "POST",
            "cache_control" : "no-cache",
            "content_type" : "text/plain",
            "content_length" : "5",
            "http_version" : "HTTP/1.1",
            "connection" : "keep-alive",
            "accept_encoding" : "gzip, deflate",
            "http_user_agent" : "PostmanRuntime/7.21.0"
          }
        }
      }
    ]
  }
}

这是我在Logstash console上看到的

{
       "@version" => "1",
    "test_string" => "Python version 1",  **<== test_string that I added in mutate filter**
     "@timestamp" => 2020-01-09T16:23:17.734Z,
           "host" => "0:0:0:0:0:0:0:1",
        "message" => "hello",    **<=== the message that I sent through POSTMAN**
        "headers" => {
           "request_path" => "/",
          "postman_token" => "9e9e45a1-d6d2-445ca-9f8f-5eae9dd15320",
            "http_accept" => "*/*",
              "http_host" => "localhost:8080",
         "request_method" => "POST",
          "cache_control" => "no-cache",
           "content_type" => "text/plain",
         "content_length" => "5",
           "http_version" => "HTTP/1.1",
             "connection" => "keep-alive",
        "accept_encoding" => "gzip, deflate",
        "http_user_agent" => "PostmanRuntime/7.21.0"
    }
}