问题:查询时间过长
我有一个看起来像这样的新表，有 3e6 行:
CREATE TABLE everything_crowberry (
id SERIAL PRIMARY KEY,
group_id INTEGER,
group_type group_type_name,
epub_id TEXT,
reg_user_id INTEGER,
device_id TEXT,
campaign_id INTEGER,
category_name TEXT,
instance_name TEXT,
protobuf TEXT,
UNIQUE (group_id, group_type, reg_user_id, category_name, instance_name)
);
这通常对我的上下文有意义，而且大多数查询的速度都可以接受。
但不快的是这样的查询:
analytics_staging=> explain analyze select count(distinct group_id) from everything_crowberry;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------
Aggregate (cost=392177.29..392177.30 rows=1 width=4) (actual time=8909.698..8909.699 rows=1 loops=1)
-> Seq Scan on everything_crowberry (cost=0.00..384180.83 rows=3198583 width=4) (actual time=0.461..6347.272 rows=3198583 loops=1)
Planning time: 0.063 ms
Execution time: 8909.730 ms
(4 rows)
Time: 8910.110 ms
analytics_staging=> select count(distinct group_id) from everything_crowberry;
count
-------
481
Time: 8736.364 ms
我确实在 group_id 上创建了一个索引，但是虽然该索引用于 WHERE 子句，但它并没有在上面使用。所以我得出结论，我误解了 postgres 如何使用索引。请注意(查询结果)有不到 500 个不同的 group_id。
CREATE INDEX everything_crowberry_group_id ON everything_crowberry(group_id);
我有什么误解或如何使这个特定查询更快的指示吗？
更新
为了帮助解决评论中提出的问题，我在此处添加了建议的更改。对于 future 的读者，我提供了详细信息以更好地理解这是如何调试的。
我注意到大部分时间都花在了初始聚合上。
序列扫描
关闭 seqscan 会使情况变得更糟:
analytics_staging=> set enable_seqscan = false;
analytics_staging=> explain analyze select count(distinct group_id) from everything_crowberry;
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------------------
Aggregate (cost=444062.28..444062.29 rows=1 width=4) (actual time=38927.323..38927.323 rows=1 loops=1)
-> Bitmap Heap Scan on everything_crowberry (cost=51884.99..436065.82 rows=3198583 width=4) (actual time=458.252..36167.789 rows=3198583 loops=1)
Heap Blocks: exact=35734 lossy=316446
-> Bitmap Index Scan on everything_crowberry_group (cost=0.00..51085.35 rows=3198583 width=0) (actual time=448.537..448.537 rows=3198583 loops=1)
Planning time: 0.064 ms
Execution time: 38927.971 ms
Time: 38930.328 ms
哪里可以使情况变得更糟
限制为一组非常小的组 ID 会使情况变得更糟，而我可能认为对一组较小的事物进行计数会更容易。
analytics_staging=> explain analyze select count(distinct group_id) from everything_crowberry WHERE group_id > 380;
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------
Aggregate (cost=385954.43..385954.44 rows=1 width=4) (actual time=13438.422..13438.422 rows=1 loops=1)
-> Bitmap Heap Scan on everything_crowberry (cost=18742.95..383451.68 rows=1001099 width=4) (actual time=132.571..12673.233 rows=986572 loops=1)
Recheck Cond: (group_id > 380)
Rows Removed by Index Recheck: 70816
Heap Blocks: exact=49632 lossy=79167
-> Bitmap Index Scan on everything_crowberry_group (cost=0.00..18492.67 rows=1001099 width=0) (actual time=120.816..120.816 rows=986572 loops=1)
Index Cond: (group_id > 380)
Planning time: 1.294 ms
Execution time: 13439.017 ms
(9 rows)
Time: 13442.603 ms
解释(分析，缓冲)
analytics_staging=> explain(analyze, buffers) select count(distinct group_id) from everything_crowberry;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------
Aggregate (cost=392177.29..392177.30 rows=1 width=4) (actual time=7329.775..7329.775 rows=1 loops=1)
Buffers: shared hit=16283 read=335912, temp read=4693 written=4693
-> Seq Scan on everything_crowberry (cost=0.00..384180.83 rows=3198583 width=4) (actual time=0.224..4615.015 rows=3198583 loops=1)
Buffers: shared hit=16283 read=335912
Planning time: 0.089 ms
Execution time: 7329.818 ms
Time: 7331.084 ms
work_mem 太小(参见上面的 explain(analyze, buffers))
将它从默认的 4 MB 增加到 10 MB 会有所改善，从 7300 毫秒增加到 5500 毫秒左右。
更改 SQL 也有一点帮助。
analytics_staging=> EXPLAIN(analyze, buffers) SELECT group_id FROM everything_crowberry GROUP BY group_id;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------
HashAggregate (cost=392177.29..392181.56 rows=427 width=4) (actual time=4686.525..4686.612 rows=481 loops=1)
Group Key: group_id
Buffers: shared hit=96 read=352099
-> Seq Scan on everything_crowberry (cost=0.00..384180.83 rows=3198583 width=4) (actual time=0.034..4017.122 rows=3198583 loops=1)
Buffers: shared hit=96 read=352099
Planning time: 0.094 ms
Execution time: 4686.686 ms
Time: 4687.461 ms
analytics_staging=> EXPLAIN(analyze, buffers) SELECT distinct group_id FROM everything_crowberry;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------
HashAggregate (cost=392177.29..392181.56 rows=427 width=4) (actual time=5536.151..5536.262 rows=481 loops=1)
Group Key: group_id
Buffers: shared hit=128 read=352067
-> Seq Scan on everything_crowberry (cost=0.00..384180.83 rows=3198583 width=4) (actual time=0.030..4946.024 rows=3198583 loops=1)
Buffers: shared hit=128 read=352067
Planning time: 0.074 ms
Execution time: 5536.321 ms
Time: 5537.380 ms
analytics_staging=> SELECT count(*) FROM (SELECT 1 FROM everything_crowberry GROUP BY group_id) ec;
count
-------
481
Time: 4927.671 ms
创建 View 是一个重大胜利，但可能会在其他地方产生性能问题。
analytics_production=> CREATE VIEW everything_crowberry_group_view AS select distinct group_id, group_type FROM everything_crowberry;
CREATE VIEW
analytics_production=> EXPLAIN(analyze, buffers) SELECT distinct group_id FROM everything_crowberry_group_view;
QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Unique (cost=0.56..357898.89 rows=200 width=4) (actual time=0.046..1976.882 rows=447 loops=1)
Buffers: shared hit=667230 read=109291 dirtied=108 written=988
-> Subquery Scan on everything_crowberry_group_view (cost=0.56..357897.19 rows=680 width=4) (actual time=0.046..1976.616 rows=475 loops=1)
Buffers: shared hit=667230 read=109291 dirtied=108 written=988
-> Unique (cost=0.56..357890.39 rows=680 width=8) (actual time=0.044..1976.378 rows=475 loops=1)
Buffers: shared hit=667230 read=109291 dirtied=108 written=988
-> Index Only Scan using everything_crowberry_group_id_group_type_reg_user_id_catego_key on everything_crowberry (cost=0.56..343330.63 rows=2911953 width=8) (actual time=0.043..1656.409 rows=2912005 loops=1)
Heap Fetches: 290488
Buffers: shared hit=667230 read=109291 dirtied=108 written=988
Planning time: 1.842 ms
Execution time: 1977.086 ms
最佳答案
对于 group_id 中相对几个 不同的值 (每组多行) - 似乎是你的情况:
3e6 rows / under 500 distinct group_id's
要使其快速，您需要索引跳过扫描(也称为松散索引扫描)。这在 Postgres 12 之前没有实现。但是你可以通过递归查询来解决这个限制:
替换:
select count(distinct group_id) from everything_crowberry;
与:
WITH RECURSIVE cte AS (
(SELECT group_id FROM everything_crowberry ORDER BY group_id LIMIT 1)
UNION ALL
SELECT (SELECT group_id FROM everything_crowberry
WHERE group_id > t.group_id ORDER BY group_id LIMIT 1)
FROM cte t
WHERE t.group_id IS NOT NULL
)
SELECT count(group_id) FROM cte;
我使用 count(group_id)而不是稍快的 count(*)方便地消除 NULL最终递归的值 - 作为 count(<expression>)只计算非空值。
另外，group_id 是否无关紧要可以是NULL ，因为您的查询无论如何都不计算在内。
可以使用已有的索引:
CREATE INDEX everything_crowberry_group_id ON everything_crowberry(group_id);
相关:
对于 group_id 中相对许多 不同的值 (每组几行) - 或者对于小表 - 普通 DISTINCT会更快。通常在子查询中完成时最快，而不是在 count() 中添加子句:
SELECT count(group_id) -- or just count(*) to include possible NULL value
FROM (SELECT DISTINCT group_id FROM everything_crowberry) sub;
关于sql - 快速计算不同列值的方法(使用索引？)，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/57558942/