celery 节拍因资源错误而立即停止

Question

celery beat 的日志是这样的，最后一行刚刚停止，不再继续和恢复。

[2018-08-20 11:20:59,002: INFO/MainProcess] Scheduler: Sending due task check result delays every 10sec (notify_delay)
[2018-08-20 11:21:00,000: INFO/MainProcess] Scheduler: Sending due task load abnormal schedules (load_abnormal_schedules)
[2018-08-20 11:21:00,004: INFO/MainProcess] Scheduler: Sending due task check close schedule every 5sec (close_schedule)
[2018-08-20 11:21:05,000: INFO/MainProcess] Scheduler: Sending due task check close schedule every 5sec (close_schedule)
[2018-08-20 11:21:10,000: INFO/MainProcess] Scheduler: Sending due task check close schedule every 5sec (close_schedule)
[2018-08-20 11:21:14,002: INFO/MainProcess] Scheduler: Sending due task check result delays every 10sec (notify_delay)
[2018-08-20 11:21:15,000: INFO/MainProcess] Scheduler: Sending due task load abnormal schedules (load_abnormal_schedules)
[2018-08-20 11:21:15,003: INFO/MainProcess] Scheduler: Sending due task check close schedule every 5sec (close_schedule)
[2018-08-20 11:21:20,000: INFO/MainProcess] Scheduler: Sending due task check close schedule every 5sec (close_schedule)
[2018-08-20 11:21:25,000: INFO/MainProcess] Scheduler: Sending due task check close schedule every 5sec (close_schedule)
[2018-08-20 11:21:29,003: INFO/MainProcess] Scheduler: Sending due task check result delays every 10sec (notify_delay)

它在 docker 容器中运行。当我通过 top 检查时，它显示了很高的 CPU 百分比

120549 root      20   0  356016 150144  16388 S  23.4  1.0   3:36.33 celery

然后当我在容器内 ssh 并尝试 celery beat 命令时。最初返回以下错误

root@4a298cc9c6e2:/usr/src/app# celery -A ghost beat -l info --pidfile=
celery beat v4.2.0 (windowlicker) is starting.

__    -    ... __   -        _
LocalTime -> 2018-08-20 11:32:51
Configuration ->
    . broker -> amqp://ghost:**@ghost-rabbitmq:5672/ghost
    . loader -> celery.loaders.app.AppLoader
    . scheduler -> celery.beat.PersistentScheduler
    . db -> celerybeat-schedule
    . logfile -> [stderr]@%INFO
    . maxinterval -> 5.00 minutes (300s)
[2018-08-20 11:32:51,526: INFO/MainProcess] beat: Starting...
[2018-08-20 11:32:51,535: ERROR/MainProcess] Removing corrupted schedule file 'celerybeat-schedule': error(11, 'Resource temporarily unavailable')
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/kombu/utils/objects.py", line 42, in __get__
    return obj.__dict__[self.__name__]
KeyError: 'scheduler'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/celery/beat.py", line 476, in setup_schedule
    self._store = self._open_schedule()
  File "/usr/local/lib/python3.6/site-packages/celery/beat.py", line 466, in _open_schedule
    return self.persistence.open(self.schedule_filename, writeback=True)
  File "/usr/local/lib/python3.6/shelve.py", line 243, in open
    return DbfilenameShelf(filename, flag, protocol, writeback)
  File "/usr/local/lib/python3.6/shelve.py", line 227, in __init__
    Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
  File "/usr/local/lib/python3.6/dbm/__init__.py", line 94, in open
    return mod.open(file, flag, mode)
_gdbm.error: [Errno 11] Resource temporarily unavailable

请注意，我只使用纯 celery 而不是 django-celery-beat

Answer 1

我亲爱的 friend ，当你启动你的 docker 容器时，celery 想要创建 celerybeat.pid 文件，如果存在，django 会引发错误。所以你应该添加这些命令来删除你的 Dockerfile 上的当前 celerybeat.pid 文件，如下所示:

COPY entrypoint.sh /code/entrypoint.sh
ENTRYPOINT ["/code/entrypoint.sh"]
RUN chmod +x /entrypoint.sh

你应该像下面这样创建一个 entrypoint.sh 文件:

#!/bin/sh
rm -rf /code/badpanty/*.pid
exec "$@"

希望对您有所帮助。