arrays - PostgreSQL fdw 表性能在 5 次相同查询后呈指数下降

Question

我有一个在 CentOS 计算机上运行的 postgres 13.3 数据库。执行这些测试时，该计算机上没有运行其他任何东西，并且在执行测试时没有其他东西访问数据库。

表 jammerdal 包含大约 500.000 行。我对包含大约 50.000 行的其他表重复了这个实验。结果是相同的，但速度减慢似乎与所使用的 fdw 表中的行数以及生成的假 id 的数量相关。

运行这个:

CREATE OR REPLACE FUNCTION jegfatterintet() RETURNS TABLE (c BIGINT)
AS $$
DECLARE
    jammerdal_ids VARCHAR[];
    area_row RECORD;
    start TIMESTAMP;
BEGIN
    SELECT INTO jammerdal_ids ARRAY_AGG('id-'||x::VARCHAR) FROM generate_series(0,25) x;

    FOR area_row IN SELECT * FROM generate_series(1,10)
        LOOP
        SELECT INTO start clock_timestamp();
        --RAISE NOTICE '%: Start %.', clock_timestamp(), area_row;
        RETURN QUERY
              SELECT COUNT(*) FROM jammerdal WHERE id = ANY(jammerdal_ids);
        --RAISE NOTICE '%: End %.', clock_timestamp(), area_row;
        RAISE NOTICE '%: Duration is %.', area_row, clock_timestamp()-start;
    END LOOP;
    RAISE NOTICE '%: All done.', clock_timestamp();
END
$$ LANGUAGE plpgsql;    

SELECT * FROM jegfatterintet();

产生以下输出:

CREATE FUNCTION
NOTICE:  (1): Duration is 00:00:00.019555.
NOTICE:  (2): Duration is 00:00:00.001271.
NOTICE:  (3): Duration is 00:00:00.001089.
NOTICE:  (4): Duration is 00:00:00.00118.
NOTICE:  (5): Duration is 00:00:00.001035.
NOTICE:  (6): Duration is 00:00:02.954527.
NOTICE:  (7): Duration is 00:00:02.871185.
NOTICE:  (8): Duration is 00:00:02.812426.
NOTICE:  (9): Duration is 00:00:02.777037.
NOTICE:  (10): Duration is 00:00:02.90708.
NOTICE:  2021-09-07 11:21:53.577115+00: All done.
 c 
---
 0
 0
 0
 0
 0
 0
 0
 0
 0
 0
(10 rows)

请注意，从第 6 步开始，持续时间如何突然从低于 0.01 秒攀升至近 3 秒。

只有当 jammerdal 是外部 (fdw) 表时才会发生这种情况，而当它是本地表时则不会发生这种情况。而且只有在使用 ids 数组时才会发生。

如果我将函数更改为:

CREATE OR REPLACE FUNCTION jegfatterintet() RETURNS TABLE (c BIGINT)
AS $$
DECLARE
    jammerdal_ids VARCHAR[];
    area_row RECORD;
    start TIMESTAMP;
BEGIN
    SELECT INTO jammerdal_ids ARRAY_AGG('id-'||x::VARCHAR) FROM generate_series(0,25) x;

    FOR area_row IN SELECT * FROM generate_series(1,10)
        LOOP
        SELECT INTO start clock_timestamp();
        --RAISE NOTICE '%: Start %.', clock_timestamp(), area_row;
        RETURN QUERY
              SELECT COUNT(*) FROM jammerdal WHERE id IN ('id-0', 'id-1', 'id-2', 'id-3', 'id-4', 'id-5', 'id-6', 'id-7', 'id-8', 'id-9', 'id-10', 'id-11', 'id-12', 'id-13', 'id-14', 'id-15', 'id-16', 'id-17', 'id-18', 'id-19', 'id-20', 'id-21', 'id-22', 'id-23', 'id-24', 'id-25'); --id = ANY(jammerdal_ids);
        --RAISE NOTICE '%: End %.', clock_timestamp(), area_row;
        RAISE NOTICE '%: Duration is %.', area_row, clock_timestamp()-start;
    END LOOP;
    RAISE NOTICE '%: All done.', clock_timestamp();
END
$$ LANGUAGE plpgsql;

SELECT * FROM jegfatterintet();

输出变为:

CREATE FUNCTION
NOTICE:  (1): Duration is 00:00:00.028254.
NOTICE:  (2): Duration is 00:00:00.001768.
NOTICE:  (3): Duration is 00:00:00.001512.
NOTICE:  (4): Duration is 00:00:00.001426.
NOTICE:  (5): Duration is 00:00:00.001523.
NOTICE:  (6): Duration is 00:00:00.001389.
NOTICE:  (7): Duration is 00:00:00.001363.
NOTICE:  (8): Duration is 00:00:00.001364.
NOTICE:  (9): Duration is 00:00:00.001466.
NOTICE:  (10): Duration is 00:00:00.001454.
NOTICE:  2021-09-07 11:25:46.635762+00: All done.
 c 
---
 0
 0
 0
 0
 0
 0
 0
 0
 0
 0
(10 rows)

谁能给我解释一下吗？

编辑:

explain (ANALYZE, BUFFERS) select id from jammerdal where id = any('{id-0,id-1,id-2,id-3,id-4,id-5,id-6,id-7,id-8,id-9,id-10,id-11,id-12,id-13,id-14,id-15,id-16,id-17,id-18,id-19,id-20,id-21,id-22,id-23,id-24,id-25}');
                                                     QUERY PLAN                                                      
---------------------------------------------------------------------------------------------------------------------
 Foreign Scan on jammerdal  (cost=100.00..62578.95 rows=26 width=37) (actual time=0.718..0.719 rows=0 loops=1)
 Planning Time: 0.153 ms
 Execution Time: 1.101 ms
(3 rows)

还有一个 EXPLAIN 在“外部”数据库中运行:

explain (ANALYZE, BUFFERS) select id from jammerdal where id = any('{id-0,id-1,id-2,id-3,id-4,id-5,id-6,id-7,id-8,id-9,id-10,id-11,id-12,id-13,id-14,id-15,id-16,id-17,id-18,id-19,id-20,id-21,id-22,id-23,id-24,id-25}');
                                                                                        QUERY PLAN                                                                                        
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Index Only Scan using jammerdal_pkey on jammerdal  (cost=0.42..219.41 rows=26 width=37) (actual time=0.286..0.290 rows=0 loops=1)
   Index Cond: (id = ANY ('{id-0,id-1,id-2,id-3,id-4,id-5,id-6,id-7,id-8,id-9,id-10,id-11,id-12,id-13,id-14,id-15,id-16,id-17,id-18,id-19,id-20,id-21,id-22,id-23,id-24,id-25}'::text[]))
   Heap Fetches: 0
   Buffers: shared hit=81
 Planning:
   Buffers: shared hit=205
 Planning Time: 2.111 ms
 Execution Time: 0.423 ms
(8 rows)

顺便说一句

ANALYZE jammerdal;

没有效果。

编辑2: 问题显然是 fdw 表不使用 id 上的索引...

ALTER SERVER testdb  OPTIONS (ADD use_remote_estimate 'true');

成功了!

Answer 1

由于查询位于函数中，PostgreSQL 会缓存执行计划。这是根据特殊的启发式进行的:

• 对于前五次执行，PostgreSQL 会生成一个使用实际参数值 (jammerdal_ids) 的“自定义计划”

• 在第六次执行时，PostgreSQL 检查“通用计划”(忽略参数值)是否估计也能执行

• 如果是，则从第六次执行开始使用通用计划以节省计划时间

就您而言，通用计划显然很糟糕。

由于您没有显示EXPLAIN (ANALYZE, BUFFERS)输出，我们只能猜测原因。但一个好的猜测是您忘记ANALYZE 外部表并且统计数据很差。所以这样做

ANALYZE jammerdal;

您应该会注意到改进。

(请注意，PostgreSQL 不会自动收集外部表的统计信息。)