mysql - INSERT INTO ndbtable SELECT FROM table 在 MySQL Cluster 中使用了荒谬的内存量

Question

如果我将 300,000 行从表(innodb 或 ndb)移动到 ndb 表中，如下所示:

INSERT INTO ndbtable2
SELECT a, b,IFNULL(c,UTC_TIMESTAMP()),CASE WHEN b = 'X' THEN e ELSE d END,f
FROM ndbtable1;

Query OK, 308372 rows affected (5 min 12.59 sec)
Records: 308372  Duplicates: 0  Warnings: 0

ndb 使用的数据内存越来越大，直到完成为止。之前/峰值/之后如下

ndb_mgm -e "all report memoryusage"
Connected to Management Server at: fl-prd-mysql1:1186
Node 1: Data usage is 2%(5752 32K pages of total 262144)
Node 1: Index usage is 0%(2428 8K pages of total 262176)
Node 2: Data usage is 2%(5752 32K pages of total 262144)
Node 2: Index usage is 0%(2428 8K pages of total 262176)

Connected to Management Server at: fl-prd-mysql1:1186
Node 1: Data usage is 62%(164013 32K pages of total 262144)
Node 1: Index usage is 1%(3136 8K pages of total 262176)
Node 2: Data usage is 62%(164013 32K pages of total 262144)
Node 2: Index usage is 1%(3136 8K pages of total 262176)

Connected to Management Server at: fl-prd-mysql1:1186
Node 1: Data usage is 3%(10293 32K pages of total 262144)
Node 1: Index usage is 1%(4590 8K pages of total 262176)
Node 2: Data usage is 3%(10293 32K pages of total 262144)
Node 2: Index usage is 1%(4590 8K pages of total 262176)

如果我的数学正确，我会插入 10293 - 5752 = 4541 = 142 MB，但内存会激增 164013 - 5752 = 158261 = 4945 MB

此外，如果我将插入限制为 50,000 行，前后差异仅为 3MB，则峰值为 780MB。

显然，当 ndb 不为空时，这会出现问题......这里发生了什么？!

Answer 1

在没有解释的情况下，手册确实承认了如下限制

As noted elsewhere in this chapter, MySQL Cluster does not handle large transactions well; it is better to perform a number of small transactions with a few operations each than to attempt a single large transaction containing a great many operations. Among other considerations, large transactions require very large amounts of memory.

http://dev.mysql.com/doc/refman/5.5/en/mysql-cluster-limitations-transactions.html

“正如本章其他地方所述” ...我已经查看过，但尚未找到任何具体内容。

仅供引用，替代方案是:

1. 使用行 LIMIT 循环迭代(例如 10,000)
2. 如果插入空表，请考虑LOAD DATA，甚至使用innodb，最后ALTER TABLE ... ENGINE = ndbcluster(两者都不是事务性的)