node.js - MongoDB 分页聚合查找在海量数据中运行缓慢

Question

我在 MongoDB 中有一个包含超过 150,000 个文档的集合。我在 Node.js 中使用 Mongoose ODM v5.4.2 for MongoDB。在检索数据时，我使用带有 $skip 和 $limit 的聚合查找进行分页。我的代码工作正常，但在 100k 个文档之后，检索数据需要 10-15 秒。但在 $skip 和 $limit 的帮助下，我一次只显示 100 条记录。我已经为foreignField 创建了索引。但速度仍然越来越慢。

campaignTransactionsModel.aggregate([{
                    $match: {
                campaignId: new importModule.objectId(campaignData._id)
            }
                },
                {
                    $lookup: {
                        from: userDB,
                        localField: "userId",
                        foreignField: "_id",
                        as: "user"
                    },
                },
                {
                    $lookup: {
                        from: 'campaignterminalmodels',
                        localField: "terminalId",
                        foreignField: "_id",
                        as: "terminal"
                    },
                },
                {
                    '$facet': {
                        edges: [{
                                $sort: {
                                    [sortBy]: order
                                }
                            },
                            { $skip: skipValue },
                            { $limit: viewBy },
                        ]
                    }
                }
            ]).allowDiskUse(true).exec(function(err, docs) {
                console.log(docs);
            });

Answer 1

查询花费的时间更长，因为服务器从输入结果的开头(跳过阶段之前)开始扫描，以跳过给定数量的文档并设置新结果。

来自官方 MongoDB 文档:

The cursor.skip() method requires the server to scan from the beginning of the input results set before beginning to return results. As the offset increases, cursor.skip() will become slower.

您可以使用range查询来模拟与.skip()或skip stage(aggregation)类似的结果

Using Range Queries

Range queries can use indexes to avoid scanning unwanted documents, typically yielding better performance as the offset grows compared to using cursor.skip() for pagination.

Descending Order

Use this procedure to implement pagination with range queries:

• Choose a field such as _id which generally changes in a consistent direction over time and has a unique index to prevent duplicate values
• Query for documents whose field is less than the start value using the $lt and cursor.sort() operators, and
• Store the last-seen field value for the next query.

Increasing Order - Query for documents whose field is less than the start value using the $gt and cursor.sort() operators, and

假设您获得的最后一个文档具有 _id : objectid1，那么您可以查询具有 _id : {$lt : objectid1} 的文档以按降序排列文档。对于递增顺序，您可以查询具有 _id : {$gt : objectid1}

的文档

阅读关于 Range queries 的官方文档了解更多信息。