elasticsearch - 如何在Logstash ElasticSearch中使用_timestamp

Question

我试图弄清楚如何在logstash中使用_timestamp。

我试图添加到映射:

   "_timestamp" : {
      "enabled" : true,
      "path" : "@timestamp"
   },

但这没有预期的效果。我在elasticsearch-template.json文件中做到了这一点(我尝试了是否使用"store"=true):

{
  "template" : "logstash-*",
  "settings" : {
    "index.refresh_interval" : "5s"
  },
  "mappings" : {
    "_default_" : {
       "_timestamp" : {
          "enabled" : true,
          "store" : true,
          "path" : "@timestamp"
       },

       "_all" : {"enabled" : true},
       "dynamic_templates" : [ {
  .....

然后将修改后的文件添加到输出过滤器

output {
  elasticsearch_http {
    template => '/tmp/elasticsearch-template.json'
    host => '127.0.0.1'
    port=>9200
  }
}

为了确保数据库是干净的，我反复进行以下操作:

curl -XDELETE http://localhost:9200/logstash*
curl -XDELETE http://localhost:9200/_template/logstash
rm ~/.sincedb_*

然后尝试导入我的日志文件。但是由于某些原因，未设置_timestamp。

映射似乎还可以

{
  "logstash-2014.03.24" : {
    "_default_" : {
      "dynamic_templates" : [ {
        "string_fields" : {
          "mapping" : {
            "index" : "analyzed",
            "omit_norms" : true,
            "type" : "string",
            "fields" : {
              "raw" : {
                "index" : "not_analyzed",
                "ignore_above" : 256,
                "type" : "string"
              }
            }
          },
          "match" : "*",
          "match_mapping_type" : "string"
        }
      } ],
      "_timestamp" : {
        "enabled" : true,
        "store" : true,
        "path" : "@timestamp"
      },
      "properties" : {
        "@version" : {
          "type" : "string",
          "index" : "not_analyzed",
          "omit_norms" : true,
          "index_options" : "docs"
        },
        "geoip" : {
          "dynamic" : "true",
          "properties" : {
            "location" : {
              "type" : "geo_point"
            }
          }
        }
      }
    },
    "logs" : {
      "dynamic_templates" : [ {
        "string_fields" : {
          "mapping" : {
            "index" : "analyzed",
            "omit_norms" : true,
            "type" : "string",
            "fields" : {
              "raw" : {
                "index" : "not_analyzed",
                "ignore_above" : 256,
                "type" : "string"
              }
            }
          },
          "match" : "*",
          "match_mapping_type" : "string"
        }
      } ],
      "_timestamp" : {
        "enabled" : true,
        "store" : true,
        "path" : "@timestamp"
      },
      "properties" : {
        "@timestamp" : {
          "type" : "date",
          "format" : "dateOptionalTime"
        },

数据库中的文档看起来像

 {
    "_id": "Cps2Lq1nTIuj_VysOwwcWw", 
    "_index": "logstash-2014.03.25", 
    "_score": 1.0, 
    "_source": {
      "@timestamp": "2014-03-25T00:47:09.703Z", 
      "@version": "1", 
      "created": "2014-03-25 01:47:09,703", 
      "host": "macbookpro.fritz.box", 
      "message": "2014-03-25 01:47:09,703 - Starting new HTTP connection (1): localhost", 
      "path": "/Users/scharf/git/ckann/annotator-store/logs/requests.log", 
      "text": "Starting new HTTP connection (1): localhost"
    }, 
    "_type": "logs"
  }, 

为什么没有设置_timestamp ？？？

Answer 1

简而言之，它确实有效。

我测试了您的确切情况，发现的是:

当启用_source并从_timestamp中的某些路径指定_source时，
您永远不会在文档中看到_timestamp，但是，如果添加了?fields查询字符串部分，例如:
http://<localhost>:9200/es_test_logs/ESTest1/ilq4PU3tR9SeoLo794wZlg?fields=_timestamp您将获得正确的_timestamp值。

如果您在外部(在_timestamp文档中)传递_source而不是使用path，则您将照常在文档中_timestamp属性下看到_source。

如果禁用_source字段，则即使在设置为store" : true的文档中，也将看不到任何ANY属性。只有在指定?fields或构建返回这些字段的查询时，才会看到它们。