elasticsearch - 如何对聚合执行 "OR"过滤器？

标签 elasticsearch

我正在尝试获取按域分组的前 10 个文档。这 10 个文档需要有一个“crawl_date”的值，该值已经有一段时间没有被抓取或根本没有被抓取(例如空白值)。我有:

curl -XPOST 'http://localhost:9200/tester/test/_search' -d '
{
  "size": 10,
  "aggs": {
    "group_by_domain": {
            "filter": {
                "or":[
                        "term": {"crawl_date": ""},
                        "term": {"crawl_date": ""}  // how do I put a range here? e.g. <= '2014-12-31'
                    ]
            },
      "terms": {
        "field": "domain"
      }
    }
  } 
}'

我是 ES 的新手，使用的是 2.2 版。由于文档没有完全更新，我正在苦苦挣扎。

编辑: 澄清一下，我需要 10 个未被抓取或一段时间未被抓取的 url。这 10 个网址中的每一个都必须来自一个唯一的域，这样当我抓取它们时，我就不会使某人的服务器重载。

另一个编辑: 所以，我需要这样的东西(10 个唯一域中的每一个域 1 个链接):

1. www.domain1.com/page
2. www.domain2.com/url
etc...

相反，我只得到域和页数:

"buckets": [
          {
            "key": "http://www.dailymail.co.uk",
            "doc_count": 212
          },
          {
            "key": "https://sedo.com",
            "doc_count": 196
          },
          {
            "key": "http://www.foxnews.com",
            "doc_count": 118
          },
          {
            "key": "http://data.worldbank.org",
            "doc_count": 117
          },
          {
            "key": "http://detail.1688.com",
            "doc_count": 117
          },
          {
            "key": "https://twitter.com",
            "doc_count": 112
          },
          {
            "key": "http://search.rakuten.co.jp",
            "doc_count": 104
          },
          {
            "key": "https://in.1688.com",
            "doc_count": 92
          },
          {
            "key": "http://www.abc.net.au",
            "doc_count": 87
          },
          {
            "key": "http://sport.lemonde.fr",
            "doc_count": 85
          }
        ]

“点击”仅返回 1 个域的多个页面:

"hits": [
      {
        "_index": "tester",
        "_type": "test",
        "_id": "http://www.barnesandnoble.com/w/at-the-edge-of-the-orchard-tracy-chevalier/1121908441?ean=9780525953005",
        "_score": 1,
        "_source": {
          "domain": "http://www.barnesandnoble.com",
          "crawl_date": "0001-01-01T00:00:00Z"
        }
      },
      {
        "_index": "tester",
        "_type": "test",
        "_id": "http://www.barnesandnoble.com/b/bargain-books/_/N-8qb",
        "_score": 1,
        "_source": {
          "domain": "http://www.barnesandnoble.com",
          "crawl_date": "0001-01-01T00:00:00Z"
        }
      },
      etc....

如果我尝试同时抓取那么多域，Barnes and Noble 将很快禁止我的 UA。

我需要这样的东西:

1. "http://www.dailymail.co.uk/page/text.html",
2. "https://sedo.com/another/page"
3. "http://www.barnesandnoble.com/b/bargain-books/_/N-8qb"
4. "http://www.starbucks.com/homepage/"
etc.

最佳答案

使用聚合

如果你想使用聚合，我建议使用术语聚合来从你的结果集中删除重复项，作为子聚合，我会使用 top_hits aggregation ，它为您提供每个域的聚合文档的最佳匹配(默认情况下，域内每个文档的分数应该相同。)

因此查询将如下所示:

POST sites/page/_search
{
  "size": 0,
  "aggs": {
    "filtered_domains": {
      "filter": {
        "bool": {
          "should": [
            {
              "bool": {
                "must_not": {
                  "exists": {
                    "field": "crawl_date"
                  }
                }
              }
            },
            {
              "range": {
                "crawl_date": {
                  "lte": "2016-01-01"
                }
              }
            }
          ]
        }
      },
      "aggs": {
        "domains": {
          "terms": {
            "field": "domain",
            "size": 10
          },
          "aggs": {
            "pages": {
              "top_hits": {
                "size": 1
              }
            }
          }
        }
      }
    }
  }
}

给你这样的结果

"aggregations": {
"filtered_domains": {
  "doc_count": 3,
  "domains": {
    "doc_count_error_upper_bound": 0,
    "sum_other_doc_count": 0,
    "buckets": [
      {
        "key": "barnesandnoble.com",
        "doc_count": 2,
        "pages": {
          "hits": {
            "total": 2,
            "max_score": 1,
            "hits": [
              {
                "_index": "test",
                "_type": "page",
                "_id": "barnesandnoble.com/test2.html",
                "_score": 1,
                "_source": {
                  "crawl_date": "1982-05-16",
                  "domain": "barnesandnoble.com"
                }
              }
            ]
          }
        }
      },
      {
        "key": "starbucks.com",
        "doc_count": 1,
        "pages": {
          "hits": {
            "total": 1,
            "max_score": 1,
            "hits": [
              {
                "_index": "test",
                "_type": "page",
                "_id": "starbucks.com/index.html",
                "_score": 1,
                "_source": {
                  "crawl_date": "1982-05-16",
                  "domain": "starbucks.com"
                }
              }
            ]
          }
        }
      }
    ]
  }
}

使用父/子聚合

如果您可以更改索引结构，我建议您创建一个具有父/子关系或嵌套文档的索引。

如果这样做，您可以选择 10 个不同的域并检索此 url 的一个(或多个)特定页面。

让我给你看一个父/子的例子(如果你使用感觉，你应该能够只复制粘贴):

首先为文档生成映射:

PUT /sites 
{
  "mappings": {
    "domain": {},
    "page": {
      "_parent": {
        "type": "domain" 
      },
      "properties": {
        "crawl_date": {
          "type": "date"
        }
      }
    }
  }
}

插入一些文件

PUT sites/domain/barnesandnoble.com
{}
PUT sites/domain/starbucks.com
{}
PUT sites/domain/dailymail.co.uk
{}
POST /sites/page/_bulk
{ "index": { "_id": "barnesandnoble.com/test.html", "parent":    "barnesandnoble.com" }}
{ "crawl_date": "1982-05-16" }
{ "index": { "_id": "barnesandnoble.com/test2.html", "parent": "barnesandnoble.com" }}
{ "crawl_date": "1982-05-16" }
{ "index": { "_id": "starbucks.com/index.html", "parent": "starbucks.com" }}
{ "crawl_date": "1982-05-16" }
{ "index": { "_id": "dailymail.co.uk/index.html", "parent": "dailymail.co.uk" }}
{}

搜索要抓取的 url

POST /sites/domain/_search
{
  "query": {
    "has_child": {
      "type": "page",
      "query": {
        "bool": {
          "filter": {
            "bool": {
              "should": [
                {
                  "bool": {
                    "must_not": {
                      "exists": {
                        "field": "crawl_date"
                      }
                    }  
                  }
                },
                {
                  "range": {
                    "crawl_date": {
                      "lte": "2016-01-01"
                    }
                  }     
                }]

            }

          }
        }
      },
      "inner_hits": {
        "size": 1
      }
    }
  }
}

我们对父类型执行 has_child 查询，因此仅接收父类型的不同 url。要获取特定页面，我们必须添加 inner_hits query ，它为我们提供了导致父类型中命中的子文档。 如果将 inner_hits 大小设置为 1，则每个域只能获得一个页面。 您甚至可以在 inner_hits 查询中添加排序...例如，您可以按 crawl_date 排序。 ;)

以上搜索为您提供以下结果:

"hits": [
  {
    "_index": "sites",
    "_type": "domain",
    "_id": "starbucks.com",
    "_score": 1,
    "_source": {},
    "inner_hits": {
      "page": {
        "hits": {
          "total": 1,
          "max_score": 1.9664046,
          "hits": [
            {
              "_index": "sites",
              "_type": "page",
              "_id": "starbucks.com/index.html",
              "_score": 1.9664046,
              "_routing": "starbucks.com",
              "_parent": "starbucks.com",
              "_source": {
                "crawl_date": "1982-05-16"
              }
            }
          ]
        }
      }
    }
  },
  {
    "_index": "sites",
    "_type": "domain",
    "_id": "dailymail.co.uk",
    "_score": 1,
    "_source": {},
    "inner_hits": {
      "page": {
        "hits": {
          "total": 1,
          "max_score": 1.9664046,
          "hits": [
            {
              "_index": "sites",
              "_type": "page",
              "_id": "dailymail.co.uk/index.html",
              "_score": 1.9664046,
              "_routing": "dailymail.co.uk",
              "_parent": "dailymail.co.uk",
              "_source": {}
            }
          ]
        }
      }
    }
  },
  {
    "_index": "sites",
    "_type": "domain",
    "_id": "barnesandnoble.com",
    "_score": 1,
    "_source": {},
    "inner_hits": {
      "page": {
        "hits": {
          "total": 2,
          "max_score": 1.4142135,
          "hits": [
            {
              "_index": "sites",
              "_type": "page",
              "_id": "barnesandnoble.com/test.html",
              "_score": 1.4142135,
              "_routing": "barnesandnoble.com",
              "_parent": "barnesandnoble.com",
              "_source": {
                "crawl_date": "1982-05-16"
              }
            }
          ]
        }
      }
    }
  }
]

最后，让我注意一件事。父/子关系在查询时的成本很小。如果这对您的用例来说不是问题，我会选择此解决方案。

关于elasticsearch - 如何对聚合执行 "OR"过滤器？，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/36110252/

上一篇：Elasticsearch 升级 2.3.1 Nest 客户端 Raw String

下一篇：ElasticSearch:Jest vs Rest vs TransportClient vs NodeClient

相关文章：

elasticsearch - Elasticsearch -如何组合多个must子句？

elasticsearch - 如何为日志设置当前时间(-zone)？

elasticsearch - Elasticsearch 中的简单分组

google-app-engine - GCP内部负载平衡器

elasticsearch - 子聚合或基数聚合中的桶选择器

每次重启时elasticsearch数据都会增加和重复

java - Elasticsearch 遇到 fatal error 并在“问题帧 : # J org. apache.lucene.index.SegmentTermEnum.next()Z”上重新启动

c# - ElasticSearch .net GeoDistance过滤器不起作用

elasticsearch - 错误无法定位插件类型 [对于 RollingFile 和 TimeBasedTriggeringPolicy]

search - 提升完成建议器的输入字段

©2024 IT工具网  联系我们