sql - 在 elasticsearch 中从 postgresql 对分区进行排名

Question

我们面临着将大型数据集从 postgres(备份或其他)迁移到 elasticsearch 的问题。

我们有类似这样的架构

+---------------+--------------+------------+-----------+
|    user_id    |  created_at  |  latitude  | longitude |
+---------------+--------------+------------+-----------+
|       5       |  23.1.2015   |    12.49   |    20.39  |
+---------------+--------------+------------+-----------+
|       2       |  23.1.2015   |    12.42   |    20.32  |
+---------------+--------------+------------+-----------+
|       2       |  24.1.2015   |    12.41   |    20.31  |
+---------------+--------------+------------+-----------+
|       5       |  25.1.2015   |    12.45   |    20.32  |
+---------------+--------------+------------+-----------+
|       1       |  23.1.2015   |    12.43   |    20.34  |
+---------------+--------------+------------+-----------+
|       1       |  24.1.2015   |    12.42   |    20.31  |
+---------------+--------------+------------+-----------+

而且我们能够通过 created_at 找到最新的位置，这要归功于 SQL 中的 rank 函数

... WITH locations AS ( 
select user_id, lat, lon, rank() over (partition by user_id order by created_at) as r
FROM locations)
SELECT user_id, lat, lon FROM locations WHERE r = 1

并且结果仅为每个用户最新创建的位置:

+---------------+--------------+------------+-----------+
|    user_id    |  created_at  |  latitude  | longitude |
+---------------+--------------+------------+-----------+
|       2       |  24.1.2015   |    12.41   |    20.31  |
+---------------+--------------+------------+-----------+
|       5       |  25.1.2015   |    12.45   |    20.32  |
+---------------+--------------+------------+-----------+
|       1       |  24.1.2015   |    12.42   |    20.31  |
+---------------+--------------+------------+-----------+

将数据导入elasticsearch后，我们的文档模型如下所示:

{
   "location" : { "lat" : 12.45, "lon" : 46.84 },
   "user_id"  : 5,
   "created_at" : "2015-01-24T07:55:20.606+00:00"
}
etc...

我正在 elasticsearch 查询中寻找此 SQL 查询的替代方案，我认为这一定是可能的，但我还没有找到。

Answer 1

您可以使用 field collapsing 结合 inner_hits 来实现这一点。

{
    "collapse": {
        "field": "user_id",
        "inner_hits": {
            "name": "order by created_at",
            "size": 1,
            "sort": [
                {
                    "created_at": "desc"
                }
            ]
        }
    },
}

详细文章:https://blog.francium.tech/sql-window-function-partition-by-in-elasticsearch-c2e3941495b6