python - Celery重启丢失定时任务

Question

我使用 Celery 安排将来发送电子邮件。我将任务放在 celery 中，并在将来有时设置了 apply_async() 和 ETA。

当我查看 flower 时，我看到为 future 安排的所有任务都处于已接收状态。

如果我重新启动 celery，所有任务都将消失。为什么他们不见了？

我使用 Redis 作为代理。

编辑1

在我发现的文档中:

If a task is not acknowledged within the Visibility Timeout the task will be redelivered to another worker and executed.

This causes problems with ETA/countdown/retry tasks where the time to execute exceeds the visibility timeout; in fact if that happens it will be executed again, and again in a loop.

So you have to increase the visibility timeout to match the time of the longest ETA you are planning to use.

Note that Celery will redeliver messages at worker shutdown, so having a long visibility timeout will only delay the redelivery of ‘lost’ tasks in the event of a power failure or forcefully terminated workers.

Periodic tasks will not be affected by the visibility timeout, as this is a concept separate from ETA/countdown.

You can increase this timeout by configuring a transport option with the same name:

BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 43200}

The value must be an int describing the number of seconds.

但我的任务的 ETA 可以用月或年来衡量。

编辑 2

这是我在输入时得到的结果:

$ celery -A app inspect scheduled

{u'priority': 6, u'eta': u'2015-11-22T11:53:00-08:00', u'request': {u'args': u'(16426,)', u'time_start': None, u'name': u'core.tasks.action_du e', u'delivery_info': {u'priority': 0, u'redelivered': None, u'routing_key': u'celery', u'exchange': u'celery'}, u'hostname': u'celery@app.myplanmap.com', u'ack nowledged': False, u'kwargs': u'{}', u'id': u'8ac59984-f8d0-47ae-ac9e-c4e3ea9c4a c6', u'worker_pid': None}}

如果你仔细观察，任务还没有被确认，所以它应该在 celery 重启后留在 redis 中，对吧？

Answer 1

你必须使用 RabbitMq 而不是 redis。

RabbitMQ 功能齐全、稳定、耐用且易于安装。这是生产环境的绝佳选择。

Redis 也是功能完备的，但在突然终止或电源故障的情况下更容易丢失数据。

使用 rabbit mq，您在重启时丢失消息的问题必须解决。