mysql - 无法关闭高负载 docker mysql

Question

我有一个脚本，每 3 秒启动一个 docker，直到有 20 个 docker 启动。之后，当一个完成时，将启动另一个 docker。

在 docker 中，我启动了一个 bash 脚本，它将启动 apache2、mysql 和一个节点脚本。一段时间后，节点脚本结束，docker 关闭。

但有时(完全随机)，docker 会永远卡在那里而不是关闭。我什至无法使用 docker stop 或 docker kill 从主机关闭它。如果我尝试重新启动主机，主机将不会完全关闭。我检查了 docker 的统计数据，而 docker 几乎保持了 0% 的 cpu 使用率。

尝试在 bash 脚本中手动关闭每个进程，我看到是 mysql 不会关闭。我尝试了 sudo service mysql stop 和 sudo mysqladmin shutdown 但都失败了。

做一个docker top，我可以看到唯一剩下的进程是当前正在运行的mysql

如果这有帮助，我使用此命令启动 docker docker run --rm test01/bin/bash/home/gps/entry_point.sh 5811fafb27517f523d1b96e5 http://10.1.5.41:8889主人

我的主机和 docker 都是 debian jessie，大部分时间都运行良好。

我不知道这是否与问题有关，但是当我在主机上执行 dmesg 时，我收到了很多这样的消息:

[ 4197.073458] docker0: port 7(veth27ee643) entered disabled state
[ 4197.764473] aufs au_opts_verify:1570:dockerd[533]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 4197.809072] aufs au_opts_verify:1570:dockerd[533]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 4197.855061] aufs au_opts_verify:1570:dockerd[533]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 4197.856599] device veth0ef0a9b entered promiscuous mode
[ 4197.857560] IPv6: ADDRCONF(NETDEV_UP): veth0ef0a9b: link is not ready
[ 4197.858392] docker0: port 7(veth0ef0a9b) entered forwarding state
[ 4197.859215] docker0: port 7(veth0ef0a9b) entered forwarding state
[ 4198.019272] docker0: port 7(veth0ef0a9b) entered disabled state
[ 4198.119297] IPv6: ADDRCONF(NETDEV_CHANGE): veth0ef0a9b: link becomes ready
[ 4198.120103] docker0: port 7(veth0ef0a9b) entered forwarding state
[ 4198.120873] docker0: port 7(veth0ef0a9b) entered forwarding state
[ 4213.166227] docker0: port 7(veth0ef0a9b) entered forwarding state
[ 4523.921860] docker0: port 3(veth65e0351) entered disabled state
[ 4523.973714] docker0: port 3(veth65e0351) entered disabled state
[ 4523.974654] device veth65e0351 left promiscuous mode
[ 4523.975105] docker0: port 3(veth65e0351) entered disabled state
[ 4525.079938] aufs au_opts_verify:1570:dockerd[533]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 4525.136297] aufs au_opts_verify:1570:dockerd[533]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 4525.196855] aufs au_opts_verify:1570:dockerd[801]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 4525.219241] device veth1658809 entered promiscuous mode
[ 4525.219777] IPv6: ADDRCONF(NETDEV_UP): veth1658809: link is not ready
[ 4525.220241] docker0: port 3(veth1658809) entered forwarding state
[ 4525.220720] docker0: port 3(veth1658809) entered forwarding state
[ 4525.221500] docker0: port 3(veth1658809) entered disabled state
[ 4525.494605] IPv6: ADDRCONF(NETDEV_CHANGE): veth1658809: link becomes ready
[ 4525.495218] docker0: port 3(veth1658809) entered forwarding state
[ 4525.495727] docker0: port 3(veth1658809) entered forwarding state
[ 4540.549216] docker0: port 3(veth1658809) entered forwarding state

知道是什么导致了这个问题吗？

编辑:不确定这是否有帮助，但是执行 sudo journalctl -fu docker.service 我得到了这些日志

Oct 27 09:03:32 Server1 dockerd[475]: time="2016-10-27T09:03:32.882004522-04:00" level=error msg="containerd: notify OOM events" error="cgroup path for memory not found"
Oct 27 09:03:39 Server1 dockerd[475]: time="2016-10-27T09:03:39.56717629-04:00" level=error msg="containerd: notify OOM events" error="cgroup path for memory not found"
Oct 27 09:03:45 Server1 dockerd[475]: time="2016-10-27T09:03:45.348335753-04:00" level=error msg="containerd: notify OOM events" error="cgroup path for memory not found"
Oct 27 09:03:52 Server1 dockerd[475]: time="2016-10-27T09:03:52.016592953-04:00" level=error msg="containerd: notify OOM events" error="cgroup path for memory not found"
Oct 27 09:05:12 Server1 dockerd[475]: time="2016-10-27T09:05:12.142848267-04:00" level=error msg="containerd: notify OOM events" error="cgroup path for memory not found"
Oct 27 09:15:35 Server1 dockerd[475]: time="2016-10-27T09:15:35.175607108-04:00" level=error msg="containerd: notify OOM events" error="cgroup path for memory not found"
Oct 27 09:23:50 Server1 dockerd[475]: time="2016-10-27T09:23:50.428877153-04:00" level=error msg="containerd: notify OOM events" error="cgroup path for memory not found"
Oct 27 09:26:29 Server1 dockerd[475]: time="2016-10-27T09:26:29.223240463-04:00" level=error msg="containerd: notify OOM events" error="cgroup path for memory not found"
Oct 27 09:31:56 Server1 dockerd[475]: time="2016-10-27T09:31:56.446979648-04:00" level=error msg="containerd: notify OOM events" error="cgroup path for memory not found"
Oct 27 10:37:18 Server1 dockerd[475]: time="2016-10-27T10:37:18.81258037-04:00" level=error msg="containerd: notify OOM events" error="cgroup path for memory not found"

Answer 1

请试试这个:

sudo chgrp mysql /etc/mysql/debian.cnf; sudo chmod 640 /etc/mysql/debian.cnf
nano /etc/systemd/system/mysql.service
[service]
ExecStop=/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf shutdown
sudo chmod 755 /etc/systemd/system/mysql.service
sudo systemctl daemon-reload