python - django-celery redis 内存错误

Question

我正在使用 django+celery 和 redis 作为代理，我的一个任务涉及读取大约 25MB 大小的大文件并返回结果，通过该结果链接另一个任务来处理结果。

由于我对redis不熟悉，所以在这里遇到错误，求助。可能是什么问题？

[2013-06-23 22:45:41,241: ERROR/MainProcess] Unrecoverable error: MemoryError()
Traceback (most recent call last):
  File "/home/property/virtualenv/property_env/lib/python2.6/site-packages/celery/worker/__init__.py", line 363, in start
    component.start()
  File "/home/property/virtualenv/property_env/lib/python2.6/site-packages/celery/worker/consumer.py", line 395, in start
    self.consume_messages()
  File "/home/property/virtualenv/property_env/lib/python2.6/site-packages/celery/worker/consumer.py", line 480, in consume_messages
    readers[fileno](fileno, event)
  File "/home/property/virtualenv/property_env/lib/python2.6/site-packages/kombu/transport/redis.py", line 770, in handle_event
    self._callbacks[queue](message)
  File "/home/property/virtualenv/property_env/lib/python2.6/site-packages/kombu/transport/virtual/__init__.py", line 479, in _callback
    self.qos.append(message, message.delivery_tag)
  File "/home/property/virtualenv/property_env/lib/python2.6/site-packages/kombu/transport/redis.py", line 117, in append
    dumps([message._raw, EX, RK])) \
  File "/home/property/virtualenv/property_env/lib/python2.6/site-packages/redis/client.py", line 1808, in execute
    return execute(conn, stack, raise_on_error)
  File "/home/property/virtualenv/property_env/lib/python2.6/site-packages/redis/client.py", line 1705, in _execute_transaction
    [args for args, options in cmds]))
MemoryError

不确定它是否提供任何提示，但检查 redis 服务器上的 maxmemory 设置似乎不是问题所在。

redis 127.0.0.1:6379> config get maxmemory
1) "maxmemory"
2) "3758096384"

Answer 1

看起来内存错误不是在redis端而是在client端(celery worker)

我的猜测是 worker 内存不足

您应该确保 celery 进程实际上可以将来自 redis 的结果分配到内存中。

如果在执行几个任务后发生这种情况，则意味着您没有足够的内存来处理您设置的并发性，或者您正在某处泄漏内存(从 redis 中保留对象的引用)