mysql - 大量导入导致 AWS Aurora 内存不足错误

Question

我正在尝试从 4.5 GB mysqldump 导入数据库文件存储到 AWS Aurora 数据库实例中。我的数据库中有大约 80 个表，最大的表有大约 1300 万行(其余的要小得多)。我的转储文件具有多值插入，其中 mysqldump根据我在 my.cnf 中指定的 max_allowed_pa​​cket，每个上限为 64MB 。

我最初在导入时遇到了各种问题，但我能够通过设置 Aurora 参数组选项来修复这些问题，即

max_allowed_packet = 1073741824 (1GB)
wait_timeout = 10800 (3 hours)
net_read_timeout = 10800
net_write_timeout = 10800
interactive_timeout = 10800

起初，我的 Aurora 实例是 db.t2.small (2GB RAM)，但是当我尝试导入时，例如mysql -u ... mydb < dump.sql从 EC2 实例( m3.medium ，4GB RAM)，该进程在运行 1 分钟后失败。 RDS 日志告诉我这是内存不足错误。我将 Aurora 实例提升到 db.t2.medium(4GB RAM)，但大约 20 分钟后该过程再次失败，并出现相同类型的内存不足错误。

我不想跳到下一个实例类型(15GB RAM)，但无论如何，我必须这样做是没有意义的。我一直在导入相同的 mysqldump定期将文件存储到我正在使用的 m3.medium EC2 实例上的本地 MySQL 服务器中，并且从未遇到过任何问题。导入大约需要 40 分钟。

这是我上次尝试导入时的 Aurora 错误日志:

Available memory is low. Trying to avoid OOM crash: system KB: 4050724 available KB: 101748 low-threshold KB: 202536 decline query: no tune caches: no kill query: no kill connection: no
OOM crash avoidance result: success: no num success: 0 system KB: 4050724 available KB: 111688 low-threshold KB: 202536 recovery time: 11 num declined query: 0 num killed query: 0 num killed connection: 0
OOM crash avoidance result: success: yes num success: 1 system KB: 4050724 available KB: 529464 low-threshold KB: 202536 recovery time: 24 num declined query: 0 num killed query: 0 num killed connection: 0
Available memory is low. Trying to avoid OOM crash: system KB: 4050724 available KB: 200956 low-threshold KB: 202536 decline query: no tune caches: no kill query: no kill connection: no
OOM crash avoidance result: success: yes num success: 2 system KB: 4050724 available KB: 556020 low-threshold KB: 202536 recovery time: 5 num declined query: 0 num killed query: 0 num killed connection: 0
Available memory is low. Trying to avoid OOM crash: system KB: 4050724 available KB: 170392 low-threshold KB: 202536 decline query: no tune caches: no kill query: no kill connection: no
OOM crash avoidance result: success: yes num success: 3 system KB: 4050724 available KB: 554108 low-threshold KB: 202536 recovery time: 7 num declined query: 0 num killed query: 0 num killed connection: 0
Available memory is low. Trying to avoid OOM crash: system KB: 4050724 available KB: 194900 low-threshold KB: 202536 decline query: no tune caches: no kill query: no kill connection: no
OOM crash avoidance result: success: yes num success: 4 system KB: 4050724 available KB: 554340 low-threshold KB: 202536 recovery time: 8 num declined query: 0 num killed query: 0 num killed connection: 0
Available memory is low. Trying to avoid OOM crash: system KB: 4050724 available KB: 198780 low-threshold KB: 202536 decline query: no tune caches: no kill query: no kill connection: no
OOM crash avoidance result: success: no num success: 4 system KB: 4050724 available KB: 133160 low-threshold KB: 202536 recovery time: 11 num declined query: 0 num killed query: 0 num killed connection: 0
OOM crash avoidance result: success: yes num success: 5 system KB: 4050724 available KB: 556540 low-threshold KB: 202536 recovery time: 25 num declined query: 0 num killed query: 0 num killed connection: 0
Available memory is low. Trying to avoid OOM crash: system KB: 4050724 available KB: 170224 low-threshold KB: 202536 decline query: no tune caches: no kill query: no kill connection: no
OOM crash avoidance result: success: yes num success: 6 system KB: 4050724 available KB: 579368 low-threshold KB: 202536 recovery time: 1 num declined query: 0 num killed query: 0 num killed connection: 0
Available memory is low. Trying to avoid OOM crash: system KB: 4050724 available KB: 175612 low-threshold KB: 202536 decline query: no tune caches: no kill query: no kill connection: no
<jemalloc>: Error in mmap(): err: 12, msg: Cannot allocate memory
<jemalloc>: Error in malloc(): out of memory
<jemalloc>: System-wide: MemTotal: 4050724kb, MemFree: 137440kb, Buffers: 20428kb, Cached: 62340kb, Active: 2188968kb, Dirty: 204kb, Inactive: 37228kb, Mapped: 41592kb
<jemalloc>: terminating process due to out of resources
10:58:03 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=6
max_threads=90
thread_count=5
connection_count=5
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63814 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x2ab59d623000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 2ab524642c08 thread_stack 0x40000
/rdsdbbin/oscar/bin/mysqld(my_print_stacktrace+0x2c)[0x9897ec]
/rdsdbbin/oscar/bin/mysqld(handle_fatal_signal+0x491)[0x6f0651]
/lib64/libpthread.so.0(+0xf5b0)[0x2ab5165405b0]
/lib64/libc.so.6(gsignal+0x39)[0x2ab51925cbe9]
/lib64/libc.so.6(abort+0x148)[0x2ab51925dfe8]
/rdsdbbin/oscar/lib/libjemalloc.so(malloc+0x1226)[0x2ab5160f62e6]
/rdsdbbin/oscar/bin/mysqld(my_malloc+0x25)[0x986525]
/rdsdbbin/oscar/bin/mysqld(alloc_root+0x8f)[0x9824bf]
/rdsdbbin/oscar/bin/mysqld[0x5b57a5]
/rdsdbbin/oscar/bin/mysqld(_Z10MYSQLparsePv+0xc1ce)[0x826ade]
/rdsdbbin/oscar/bin/mysqld(_Z9parse_sqlP3THDP12Parser_stateP19Object_creation_ctx+0xb5)[0x772175]
/rdsdbbin/oscar/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0xf2)[0x772502]
/rdsdbbin/oscar/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0xf43)[0x774003]
/rdsdbbin/oscar/bin/mysqld(_ZN22OscarSchedulerConsumer7consumeEjj+0xd3)[0x803963]
/rdsdbbin/oscar/bin/mysqld(_ZN22OscarSchedulerConsumer5startEv+0x98)[0x803a98]
/rdsdbbin/oscar/bin/mysqld(_ZN22OscarSchedulerConsumer11drain_queueEPv+0x6a)[0x803cda]
/lib64/libpthread.so.0(+0x7f18)[0x2ab516538f18]
/lib64/libc.so.6(clone+0x6d)[0x2ab51930bb2d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (2ab5ad400010): INSERT INTO `Documents` VALUES (478555572,150317,1321817,1,9,627609600,0,60,5471267,0,639014400,''),(478555571,150317,1321816,1,1,623980800,0,60,0,0,623980800,''),(478555575,150318,1321820,1,1,623980800,0,60,0,0,623980800,'')
Connection ID (thread ID): 6
Status: NOT_KILLED

鉴于我的具有 4GB RAM 的 EC2 实例可以正常处理导入，这肯定是配置问题。我可以尝试更改其他一些参数组选项吗？

我还尝试通过设置 binlog_format 来禁用二进制日志记录在我的数据库集群参数组中将参数设置为 OFF (按照说明 here )并重新启动实例，但是当我运行查询 select @@binlog_format 时我得到结果声明。

Answer 1

SET GLOBAL max_connections=6 # 从 90 开始将减少 [mysqld] RAM 要求。 导入后根据需要提高限制。