Dotcloud Supervisord 显示错误，但进程正在运行

Question

我的 dotcloud 设置(带有rabbitmq的django-celery)在一周前运行良好 - 进程启动正常并且日志很干净。但是，我最近重新推送了(没有更新任何代码)，现在日志显示进程无法启动，即使它们似乎正在运行。

主管日志

dotcloud@hack-default-www-0:/var/log/supervisor$ more supervisord.log
2012-06-03 10:51:51,836 CRIT Set uid to user 1000
2012-06-03 10:51:51,836 WARN Included extra file "/etc/supervisor/conf.d/uwsgi.c
onf" during parsing
2012-06-03 10:51:51,836 WARN Included extra file "/home/dotcloud/current/supervi
sord.conf" during parsing
2012-06-03 10:51:51,938 INFO RPC interface 'supervisor' initialized
2012-06-03 10:51:51,938 WARN cElementTree not installed, using slower XML parser
 for XML-RPC
2012-06-03 10:51:51,938 CRIT Server 'unix_http_server' running without any HTTP
authentication checking
2012-06-03 10:51:51,946 INFO daemonizing the supervisord process
2012-06-03 10:51:51,947 INFO supervisord started with pid 144
2012-06-03 10:51:53,128 INFO spawned: 'celerycam' with pid 159
2012-06-03 10:51:53,133 INFO spawned: 'apnsd' with pid 161
2012-06-03 10:51:53,148 INFO spawned: 'djcelery' with pid 164
2012-06-03 10:51:53,168 INFO spawned: 'uwsgi' with pid 167
2012-06-03 10:51:53,245 INFO exited: djcelery (exit status 1; not expected)
2012-06-03 10:51:53,247 INFO exited: celerycam (exit status 1; not expected)
2012-06-03 10:51:54,698 INFO spawned: 'celerycam' with pid 176
2012-06-03 10:51:54,698 INFO success: apnsd entered RUNNING state, process has s
tayed up for > than 1 seconds (startsecs)
2012-06-03 10:51:54,705 INFO spawned: 'djcelery' with pid 177
2012-06-03 10:51:54,706 INFO success: uwsgi entered RUNNING state, process has s
tayed up for > than 1 seconds (startsecs)
2012-06-03 10:51:54,731 INFO exited: djcelery (exit status 1; not expected)
2012-06-03 10:51:54,754 INFO exited: celerycam (exit status 1; not expected)
2012-06-03 10:51:56,760 INFO spawned: 'celerycam' with pid 178
2012-06-03 10:51:56,765 INFO spawned: 'djcelery' with pid 179
2012-06-03 10:51:56,790 INFO exited: celerycam (exit status 1; not expected)
2012-06-03 10:51:56,791 INFO exited: djcelery (exit status 1; not expected)
2012-06-03 10:51:59,798 INFO spawned: 'celerycam' with pid 180
2012-06-03 10:52:00,538 INFO spawned: 'djcelery' with pid 181
2012-06-03 10:52:00,565 INFO exited: celerycam (exit status 1; not expected)
2012-06-03 10:52:00,571 INFO gave up: celerycam entered FATAL state, too many st
art retries too quickly
2012-06-03 10:52:00,573 INFO exited: djcelery (exit status 1; not expected)
2012-06-03 10:52:01,575 INFO gave up: djcelery entered FATAL state, too many sta
rt retries too quickly
dotcloud@hack-default-www-0:/var/log/supervisor$

djerror 日志:

dotcloud@hack-default-www-0:/var/log/supervisor$ more djcelery_error.log
Traceback (most recent call last):
  File "hack/manage.py", line 2, in 
    from django.core.management import execute_manager
ImportError: No module named django.core.management
Traceback (most recent call last):
  File "hack/manage.py", line 2, in 
    from django.core.management import execute_manager
ImportError: No module named django.core.management
Traceback (most recent call last):
  File "hack/manage.py", line 2, in 
    from django.core.management import execute_manager
ImportError: No module named django.core.management
Traceback (most recent call last):
  File "hack/manage.py", line 2, in 
    from django.core.management import execute_manager
ImportError: No module named django.core.management
dotcloud@hack-default-www-0:/var/log/supervisor$

statusctrl 显示一些进程正在运行，但 pid 不同。另外， celery 功能似乎工作正常。消息已处理，我可以在 django 管理界面中看到正在处理的消息(dj celery cam 正在运行)。

# supervisorctl status
apnsd                            RUNNING    pid 225, uptime 0:00:44
celerycam                        RUNNING    pid 224, uptime 0:00:44
djcelery                         RUNNING    pid 226, uptime 0:00:44

Supervisord.conf 文件:

[program:djcelery]
directory = /home/dotcloud/current/
command = python hack/manage.py celeryd -E -l info -c 2
stderr_logfile = /var/log/supervisor/%(program_name)s_error.log
stdout_logfile = /var/log/supervisor/%(program_name)s.log

[program:celerycam]
directory = /home/dotcloud/current/
command = python hack/manage.py celerycam
stderr_logfile = /var/log/supervisor/%(program_name)s_error.log
stdout_logfile = /var/log/supervisor/%(program_name)s.log

http://jefurii.cafejosti.net/blog/2011/01/26/celery-in-virtualenv-with-supervisord/说问题可能是使用的python不正确，所以我在supervisord文件中明确指定了python。它现在可以工作了，但它并没有解释我在上面看到的内容以及为什么我必须在上周工作正常的情况下更改我的配置。

此外，并非所有 pid 都在排队​​:

2012-06-03 11:19:03,045 CRIT Server 'unix_http_server' running without any HTTP
authentication checking
2012-06-03 11:19:03,051 INFO daemonizing the supervisord process
2012-06-03 11:19:03,052 INFO supervisord started with pid 144
2012-06-03 11:19:04,061 INFO spawned: 'celerycam' with pid 151
2012-06-03 11:19:04,066 INFO spawned: 'apnsd' with pid 153
2012-06-03 11:19:04,085 INFO spawned: 'djcelery' with pid 155
2012-06-03 11:19:04,104 INFO spawned: 'uwsgi' with pid 156
2012-06-03 11:19:05,271 INFO success: celerycam entered RUNNING state, process h
as stayed up for > than 1 seconds (startsecs)
2012-06-03 11:19:05,271 INFO success: apnsd entered RUNNING state, process has s
tayed up for > than 1 seconds (startsecs)
2012-06-03 11:19:05,271 INFO success: djcelery entered RUNNING state, process ha
s stayed up for > than 1 seconds (startsecs)
2012-06-03 11:19:05,271 INFO success: uwsgi entered RUNNING state, process has s
tayed up for > than 1 seconds (startsecs)

状态显示 celery cam pid 未排队:

# supervisorctl status
apnsd                            RUNNING    pid 153, uptime 0:06:17
celerycam                        RUNNING    pid 150, uptime 0:06:17
djcelery                         RUNNING    pid 155, uptime 0:06:17

Answer 1

我的第一个猜测是您使用了错误的 python 二进制文件(system python，而不是 virtualenv python)，并且它导致了此错误(如下)，因为 system python 二进制文件没有安装该软件包。

  Traceback (most recent call last):
  File "hack/manage.py", line 2, in 
    from django.core.management import execute_manager
  ImportError: No module named django.core.management

您应该将supervisord.conf更改为以下内容，以确保您指向正确的python版本。

[program:djcelery]
directory = /home/dotcloud/current/
command = /home/dotcloud/env/bin/python hack/manage.py celeryd -E -l info -c 2
stderr_logfile = /var/log/supervisor/%(program_name)s_error.log
stdout_logfile = /var/log/supervisor/%(program_name)s.log

[program:celerycam]
directory = /home/dotcloud/current/
command = /home/dotcloud/env/bin/python hack/manage.py celerycam
stderr_logfile = /var/log/supervisor/%(program_name)s_error.log
stdout_logfile = /var/log/supervisor/%(program_name)s.log

Python 路径从 python 到 /home/dotcloud/env/bin/python。

我不确定为什么主管说它正在运行，但实际上并没有运行，但希望这一点小小的改变将有助于清除您的错误，并使一切恢复正常。