运行 Postgres 12.5 的本地 docker 实例(具有 4MB work_mem)。我正在实现 this pattern搜索 json 中的任意字段。目标是快速搜索并返回 JSON 列 profile。
我在物化 View 上尝试了 EXISTS 子查询:
CREATE TABLE end_user (
id varchar NOT NULL,
environment_id varchar NOT NULL,
profile jsonb NOT NULL DEFAULT '{}'::jsonb,
CONSTRAINT end_user_pkey PRIMARY KEY (environment_id, id)
);
CREATE INDEX end_user_environment_id_idx ON private.end_user USING btree (environment_id);
CREATE INDEX end_user_id_idx ON private.end_user USING btree (id);
CREATE INDEX end_user_profile_idx ON private.end_user USING gin (profile);
CREATE MATERIALIZED VIEW user_profiles AS
SELECT u.environment_id, u.id, j.key, j.value
FROM end_user u, jsonb_each_text(u.profile) j(key, value);
CREATE UNIQUE INDEX on user_profiles (environment_id, id, key);
CREATE INDEX user_profile_trgm_idx ON user_profiles using gin (value gin_trgm_ops);
我的查询是 indexed correctly以便它在几毫秒内执行超过一百万行。 ✅
select * from user_profiles
where value ilike '%auckland%' and key = 'timezone' and environment_id = 'test';
执行时间 42ms 😘
Bitmap Heap Scan on user_profiles (cost=28935.65..62591.44 rows=9659 width=65)
Recheck Cond: ((value ~~* '%auckland%'::text) AND (key = 'timezone'::text))
Filter: ((environment_id)::text = 'test'::text)
-> BitmapAnd (cost=28935.65..28935.65 rows=9659 width=0)
-> Bitmap Index Scan on user_profile_trgm_idx (cost=0.00..2923.95 rows=320526 width=0)
Index Cond: (value ~~* '%auckland%'::text)
-> Bitmap Index Scan on user_profiles_key_idx (cost=0.00..26006.62 rows=994408 width=0)
Index Cond: (key = 'timezone'::text)
但是，如果我将它与 exists 查询一起使用以建立如下所示的条件:
select * users u
where
environment_id = 'test'
and exists (
select 1 from user_profiles p
where
value ilike '%auckland%'
and key = 'timezone'
and p.id = u.id
and environment_id = 'test'
)
它的执行速度非常慢。
执行时间 17.44 秒 💀
Nested Loop (cost=62616.01..124606.45 rows=9658 width=1459) (actual time=19206.818..28444.491 rows=332572 loops=1)
Buffers: shared hit=952734 read=624101
-> HashAggregate (cost=62615.59..62707.52 rows=9193 width=15) (actual time=19205.238..19292.998 rows=332572 loops=1)
Group Key: (p.id)::text
Buffers: shared hit=373 read=246174
-> Bitmap Heap Scan on user_profiles p (cost=28935.65..62591.44 rows=9659 width=15) (actual time=278.211..18942.629 rows=332572 loops=1)
Recheck Cond: ((value ~~* '%auckland%'::text) AND (key = 'timezone'::text))
Rows Removed by Index Recheck: 17781109
Filter: ((environment_id)::text = 'test'::text)
Heap Blocks: exact=43928 lossy=197955
Buffers: shared hit=373 read=246174
-> BitmapAnd (cost=28935.65..28935.65 rows=9659 width=0) (actual time=272.626..272.629 rows=0 loops=1)
Buffers: shared hit=373 read=4291
-> Bitmap Index Scan on user_profile_trgm_idx (cost=0.00..2923.95 rows=320526 width=0) (actual time=177.577..177.577 rows=332572 loops=1)
Index Cond: (value ~~* '%auckland%'::text)
Buffers: shared hit=373 read=455
-> Bitmap Index Scan on user_profiles_key_idx (cost=0.00..26006.62 rows=994408 width=0) (actual time=92.586..92.589 rows=1000000 loops=1)
Index Cond: (key = 'timezone'::text)
Buffers: shared read=3836
-> Index Scan using end_user_id_idx on end_user u (cost=0.42..6.79 rows=1 width=1459) (actual time=0.027..0.027 rows=1 loops=332572)
Index Cond: ((id)::text = (p.id)::text)
Filter: ((environment_id)::text = 'test'::text)
Buffers: shared hit=952361 read=377927
Planning Time: 19.002 ms
Execution Time: 28497.570 ms
这很遗憾，因为如果 exists 速度快的话会很方便，因为我可以在我的应用程序代码中动态添加更多条件，并将额外条件表示为附加 exists 子句。
顺便说一句，横向连接确实可以加快速度，但我不明白为什么会有这么大的差异:
select * from users u,
lateral (
select id from user_profiles p
where
value ilike '%auckland%'
and key = 'timezone'
and environment_id = u.environment_id
and p.id = u.id
) ss
where u.environment_id = 'test';
执行时间304ms👍
Gather (cost=29936.07..91577.38 rows=9658 width=1474) (actual time=1100.824..15430.620 rows=332572 loops=1)
Workers Planned: 2
Workers Launched: 2
Buffers: shared hit=1140551 read=436286
-> Nested Loop (cost=28936.07..89611.58 rows=4024 width=1474) (actual time=602.490..14805.285 rows=110857 loops=3)
Buffers: shared hit=1140551 read=436286
-> Parallel Bitmap Heap Scan on user_profiles p (cost=28935.65..62492.84 rows=4025 width=22) (actual time=602.078..12247.891 rows=110857 loops=3)
Recheck Cond: ((value ~~* '%auckland%'::text) AND (key = 'timezone'::text))
Rows Removed by Index Recheck: 5927036
Filter: ((environment_id)::text = 'test'::text)
Heap Blocks: exact=14659 lossy=65588
Buffers: shared hit=373 read=246174
-> BitmapAnd (cost=28935.65..28935.65 rows=9659 width=0) (actual time=1087.258..1087.259 rows=0 loops=1)
Buffers: shared hit=373 read=4291
-> Bitmap Index Scan on user_profile_trgm_idx (cost=0.00..2923.95 rows=320526 width=0) (actual time=853.075..853.076 rows=332572 loops=1)
Index Cond: (value ~~* '%auckland%'::text)
Buffers: shared hit=373 read=455
-> Bitmap Index Scan on user_profiles_key_idx (cost=0.00..26006.62 rows=994408 width=0) (actual time=231.295..231.295 rows=1000000 loops=1)
Index Cond: (key = 'timezone'::text)
Buffers: shared read=3836
-> Index Scan using end_user_id_idx on end_user u (cost=0.42..6.74 rows=1 width=1459) (actual time=0.022..0.022 rows=1 loops=332572)
Index Cond: ((id)::text = (p.id)::text)
Filter: ((environment_id)::text = 'test'::text)
Buffers: shared hit=1140178 read=190112
Planning Time: 16.877 ms
Execution Time: 15461.571 ms
渴望听到关于为什么 exists 子查询如此慢的任何想法，以及我可以在此处查看的任何其他选项。
根据 Erwin 的要求进行不同计数，请注意，这是用于测试负载的虚拟数据，但其相当接近生产比率
select count(distinct environment_id) => 4
, count(distinct key) => 33
, count(distinct value) => 15M
from private.user_profiles;
按照 Erwin 的建议将工作内存增加到 16MB 后更新:
更改系统设置 work_mem 为“16MB”；
选择 pg_reload_conf();
exists 查询的执行时间为 500 毫秒，情况看起来更好。现在解释如下。
Gather (cost=3926.79..400754.43 rows=9658 width=1459) (actual time=312.213..9396.610 rows=332572 loops=1)
Workers Planned: 2
Workers Launched: 2
Buffers: shared hit=1141083 read=431918
-> Nested Loop (cost=2926.79..398788.63 rows=4024 width=1459) (actual time=155.271..8987.721 rows=110857 loops=3)
Buffers: shared hit=1141083 read=431918
-> Parallel Bitmap Heap Scan on user_profiles p (cost=2926.36..371669.88 rows=4025 width=15) (actual time=150.989..2962.870 rows=110857 loops=
Recheck Cond: (value ~~* '%auckland%'::text)
Filter: (((environment_id)::text = 'test'::text) AND (key = 'timezone'::text))
Heap Blocks: exact=82556
Buffers: shared hit=981 read=241730
-> Bitmap Index Scan on user_profile_trgm_idx (cost=0.00..2923.95 rows=320526 width=0) (actual time=243.604..243.605 rows=332572 loops=1
Index Cond: (value ~~* '%auckland%'::text)
Buffers: shared hit=828
-> Index Scan using end_user_id_idx on end_user u (cost=0.42..6.74 rows=1 width=1459) (actual time=0.054..0.054 rows=1 loops=332572)
Index Cond: ((id)::text = (p.id)::text)
Filter: ((environment_id)::text = 'test'::text)
Buffers: shared hit=1140102 read=190188
Planning Time: 9.932 ms
Execution Time: 9427.067 ms
最佳答案
Postgres 12 或更高版本中的 SQL/JSON
My answer you have been working off已过时。当前的 Postgres 版本是 2015 年 7 月的 9.4。
在 Postgres 12 中，通过 SQL/JSON 路径表达式中的正则表达式，整个设计可以大大简化。 The manual:
SQL/JSON path expressions allow matching text to a regular expression with the
like_regexfilter.
也有索引支持。废弃物化 View 。我们需要的只是您的原始表格和索引，例如:
CREATE INDEX end_user_path_ops_idx ON end_user USING GIN (profile jsonb_path_ops);
此查询与您的原始查询等效，并且可以使用索引:
SELECT *
FROM end_user u
WHERE environment_id = 'test'
AND profile @? '$.timezone ? (@ like_regex "auck" flag "i")';
db<> fiddle here
一个缺点是 SQL/JSON 路径语言需要习惯。
进一步阅读:
- Find rows containing a key in a JSONB array of records
- https://www.postgresql.org/docs/current/functions-json.html#FUNCTIONS-SQLJSON-PATH
服务器配置
在 EXPLAIN 输出的这一行中，一个更基本的问题变得明显:
Heap Blocks: exact=14659 lossy=65588
lossy 表示您没有足够的 work_mem 。你的设置显然很低。 (对于涉及数百万行表的数据库，默认设置 4 MB 太低了。)请参阅:
- “Recheck Cond:” line in query plans with a bitmap index scan
- PostgreSQL GIN index slower than GIST for pg_trgm?
很可能，您需要在服务器配置部门做更多的事情。一般来说，你的内存似乎受到限制。我看到较高的“读取”计数，这表明冷缓存和/或缓存内存缺乏或配置错误。
这个Postgres Wiki page可以帮助您入门。
关于postgresql - JSON 值的模式匹配，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/68192458/