elasticsearch - map 上的 Elasticsearch 聚合 - 每个键

Question

我有以下类型的文件。

文档1

{
  "doc": {
    "id": 1,
    "errors": {
      "e1":5,
      "e2":20,
      "e3":30
    },
    "warnings": {
      "w1":1,
      "w2":2
    }
  }
}

文档2

{
  "doc": {
    "id": 2,
    "errors": {
      "e1":10
    },
    "warnings": {
      "w1":1,
      "w2":2,
      "w3":33,
    }
  }
}

我想在一次或多次通话中获得以下总统计数据。是否可以？我尝试了各种解决方案，但当 key 已知时，所有解决方案都有效。在我的例子中， map 键(e1、e2等)是未知的。

{
  "errors": {
    "e1": 15,
    "e2": 20,
    "e3": 30
  },
  "warnings": {
    "w1": 2,
    "w2": 4,
    "w3": 33
  }
}

Answer 1

有两种解决方案，但都不是很好。我必须指出，选项 2 应该是首选方法，因为选项 1 使用实验性功能。

1。动态映射，[实验]脚本聚合

灵感来自this answer和 Scripted Metric Aggregation在 ES 文档页面，我首先将文档插入到不存在的索引(默认情况下会创建 dynamic mapping )。

注意:我在 ES 5.4 上测试了此功能，但文档表明此功能至少从 2.0 开始可用。

聚合查询结果如下:

POST /my_index/my_type/_search
{
    "size": 0,
    "query" : {
        "match_all" : {}
    },
    "aggs": {
        "errors": {
            "scripted_metric": {
                "init_script" : "params._agg.errors = [:]",
                "map_script" : "for (t in params['_source']['doc']['errors'].entrySet()) { params._agg.errors[t.key] = t.value } ",
                "combine_script" : "return params._agg.errors",
                "reduce_script": "Map res = [:] ; for (a in params._aggs) { for (t in a.entrySet()) { res[t.key] = res.containsKey(t.key) ? res[t.key] + t.value : t.value } }  return res"

            }
        },
        "warnings": {
            "scripted_metric": {
                "init_script" : "params._agg.errors = [:]",
                "map_script" : "for (t in params['_source']['doc']['warnings'].entrySet()) { params._agg.errors[t.key] = t.value } ",
                "combine_script" : "return params._agg.errors",
                "reduce_script": "Map res = [:] ; for (a in params._aggs) { for (t in a.entrySet()) { res[t.key] = res.containsKey(t.key) ? res[t.key] + t.value : t.value } }  return res"

            }
        }
    }
}

产生以下输出:

{
   ...
   "aggregations": {
      "warnings": {
         "value": {
            "w1": 2,
            "w2": 4,
            "w3": 33
         }
      },
      "errors": {
         "value": {
            "e1": 15,
            "e2": 20,
            "e3": 30
         }
      }
   }
}

如果您遵循此路径，您可能会对 params['_source'] 的 JavaDoc 感兴趣。在下面。

警告:我认为脚本聚合效率不高，为了获得更好的性能，您应该检查选项 2 或不同的数据处理引擎。

experimental 是什么意思？意思是:

This functionality is experimental and may be changed or removed completely in a future release. Elastic will take a best effort approach to fix any issues, but experimental features are not subject to the support SLA of official GA features.

考虑到这一点，我们继续选择选项 2。

2。静态嵌套映射、嵌套聚合

这里的想法是以不同的方式存储数据，并且本质上能够以不同的方式查询和聚合数据。首先，我们需要使用nested data type创建一个映射。 .

PUT /my_index_nested/
{
    "mappings": {
        "my_type": {
            "properties": {
                "errors": { 
                    "type": "nested",
                    "properties": {
                        "name": {"type": "keyword"},
                        "val": {"type": "integer"}
                    }
                },
                "warnings": { 
                    "type": "nested",
                    "properties": {
                        "name": {"type": "keyword"},
                        "val": {"type": "integer"}
                    }
                }
            }
        }
    }
}

此类索引中的文档将如下所示:

     {
        "_index": "my_index_nested",
        "_type": "my_type",
        "_id": "1",
        "_score": 1,
        "_source": {
           "errors": [
              {
                 "name": "e1",
                 "val": 5
              },
              {
                 "name": "e2",
                 "val": 20
              },
              {
                 "name": "e3",
                 "val": 30
              }
           ],
           "warnings": [
              {
                 "name": "w1",
                 "val": 1
              },
              {
                 "name": "w2",
                 "val": 2
              }
           ]
        }
     }

接下来我们需要编写聚合查询。首先我们需要使用nested aggregation ，这将允许我们查询这种特殊的嵌套数据类型。但由于我们实际上想要按 name 进行聚合，并对 val 的值求和，因此我们需要执行 sub-aggregation .

生成的查询如下(为了清楚起见，我在查询旁边添加了注释):

POST /my_index_nested/my_type/_search
{
    "size": 0,
    "aggs": {
        "errors_top": {
            "nested": {
                // declare which nested objects we want to work with
                "path": "errors"  
            },
            "aggs": {
                "errors": {
                    // what we are aggregating - different values of name
                    "terms": {"field": "errors.name"}, 
                    // sub aggregation
                    "aggs": { 
                        "error_sum": {
                            // sum all val for same name
                            "sum": {"field": "errors.val"} 
                        }
                        
                    }
                }
            }
        },
        "warnings_top": {
            // analogous to errors
        }
    }
}

该查询的输出如下:

{
   ...
   "aggregations": {
      "errors_top": {
         "doc_count": 4,
         "errors": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
               {
                  "key": "e1",
                  "doc_count": 2,
                  "error_sum": {
                     "value": 15
                  }
               },
               {
                  "key": "e2",
                  "doc_count": 1,
                  "error_sum": {
                     "value": 20
                  }
               },
               {
                  "key": "e3",
                  "doc_count": 1,
                  "error_sum": {
                     "value": 30
                  }
               }
            ]
         }
      },
      "warnings_top": {
         ...
      }
   }
}