django - Nginx、Gunicorn、Django、Celery(Redis) : upstream prematurely closed connection 502 gateway

Question

我在 Linux 服务器上使用 docker-compose 运行一个设置。两天前，我在设置中添加了 gunicorn + nginx。不幸的是，所有启动 celery 任务的 rest api 端点都停止工作(它返回 502 gateway not found)。

当我尝试在启动 celery 任务的calculate shortest path 上发送帖子表单时，502 网关返回。

Issue:
 Summary
 URL: http://192.168.0.150:8001/tspweb/calculate_shortest_paths/
 Status: 502 Bad Gateway
 Source: Network
 Address: 192.168.0.150:8001

这是来自 django 容器和 nginx 容器的日志。

tspoptimization | [2018-10-31 07:26:30 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:15)
nginx_1   | 2018/10/31 07:26:30 [error] 8#8: *9 upstream prematurely closed connection while reading response header from upstream, client: 192.168.0.103, server: localhost, request: "POST /tspweb/calculate_shortest_paths/ HTTP/1.1", upstream: "http://192.168.128.2:8001/tspweb/calculate_shortest_paths/", host: "192.168.0.150:8001", referrer: "http://192.168.0.150:8001/tspweb/warehouse_list.html"
nginx_1   | 192.168.0.103 - - [31/Oct/2018:07:26:30 +0000] "POST /tspweb/calculate_shortest_paths/ HTTP/1.1" 502 157 "http://192.168.0.150:8001/tspweb/warehouse_list.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15" "-

在添加 gunicorn + nginx 之前一切都运行良好(在没有这两个的本地系统上它运行得很好)。这意味着它不是超时问题。

我怀疑 nginx+gunicorn 不会将 POST 请求从表单“重定向”到 celery。我通过登录文件启动 celery ，这是 celery 日志文件的内容:

root@4fb6e101a85b:/opt/services/djangoapp/src# cat logmato.log
[2018-10-31 07:12:04,400: INFO/MainProcess] Connected to.    redis://redis:6379//
[2018-10-31 07:12:04,409: INFO/MainProcess] mingle: searching for neighbors
[2018-10-31 07:12:05,430: INFO/MainProcess] mingle: all alone
[2018-10-31 07:12:05,446: WARNING/MainProcess] /usr/local/lib/python3.6/site-packages/celery/fixups/django.py:200:     UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
 warnings.warn('Using settings.DEBUG leads to a memory leak, never '
 [2018-10-31 07:12:05,446: INFO/MainProcess] celery@4fb6e101a85b ready.
 [2018-10-31 07:14:07,802: INFO/MainProcess] Connected to redis://redis:6379//
 [2018-10-31 07:14:07,813: INFO/MainProcess] mingle: searching for neighbors
[2018-10-31 07:14:08,835: INFO/MainProcess] mingle: all alone
[2018-10-31 07:14:08,853: WARNING/MainProcess]/usr/local/lib/python3.6/site-packages/celery/fixups/django.py:200:          UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
warnings.warn('Using settings.DEBUG leads to a memory leak, never
2018-10-31 07:14:08,853: INFO/MainProcess] celery@4fb6e101a85b ready.

从日志中可以看出，celery worker 没有启动单个任务，这意味着问题不在 celery 或 redis 中，而是在 (nginx-gunicorn-django-celery) 之间的某个地方。

这是我的 docker-compose 文件:

version: '3'
services:
  db:
    image: postgres
    volumes:
      - postgres_data:/var/lib/postgresql/data
    networks:  # <-- connect to the bridge
      - database_network
  redis:
    image: "redis:alpine"
    expose:
      - "5672"
  django:
    build: .
    restart: always
    container_name: tspoptimization
    volumes:
      - .:/opt/services/djangoapp/src
      - static_volume:/opt/services/djangoapp/src/tspweb/static  # <-- bind the static volume
      - media_volume:/opt/services/djangoapp/src/tspweb/media  # <-- bind the media volume
    depends_on:
      - db
      - redis
    networks:
    - nginx_network
    - database_network
  celery:
    build: .
    command: celery -A tspoptimization worker -l info
    volumes:
      - .:/code
    depends_on:
      - db
      - redis
      - django
    links:
      - redis
  nginx:
    image: nginx:latest
    ports:
     - 8001:80
    volumes:
     - ./config/nginx/conf.d:/etc/nginx/conf.d
     - static_volume:/opt/services/djangoapp/src/tspweb/static  # <-- bind the static volume
     - media_volume:/opt/services/djangoapp/src/tspweb/media  # <-- bind the media volume
    depends_on:
     - django
    networks:
     - nginx_network
networks:
  nginx_network:
    driver: bridge
  database_network:  # <-- add the bridge
    driver: bridge
volumes:
  postgres_data:
  static_volume:
  media_volume:

这是 nginx 配置文件:

upstream hello_server {
    server django:8001;
}

server {

    listen 80;
    server_name localhost;

    location / {
        # everything is passed to Gunicorn

        proxy_pass http://hello_server;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_redirect off;
    }

    location /tspweb/static/ {
        alias /opt/services/djangoapp/src/tspweb/static/;
    }

    location /tspweb/media/ {
        alias /opt/services/djangoapp/src/tspweb/media/;
    }
}

我的 Django 设置:

DEBUG = True
ALLOWED_HOSTS = ['*']
CELERY_BROKER_URL = 'redis://redis:6379'
STATIC_URL = '/tspweb/static/'
STATIC_ROOT = os.path.join(os.path.dirname(os.path.dirname(BASE_DIR)), '/tspweb/static')
MEDIA_URL = '/tspweb/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'tspweb/media')

最后，dockerfile:

FROM python:3.6

RUN mkdir -p /opt/services/djangoapp/src
WORKDIR /opt/services/djangoapp/src
ADD . /opt/services/djangoapp/src

EXPOSE 8001
RUN pip install -r requirements.txt
RUN python manage.py collectstatic --no-input
CMD ["gunicorn", "--bind", ":8001", "tspoptimization.wsgi"]

对解决这个问题有什么帮助吗？

Answer 1

我单独解决了这个问题，所以这是答案:

Redis & Celery 必须在 docker 与 nginx-network 和 db-network 创建的同一个虚拟网络中。

这是正在运行的 docker-compose 文件，任务已正确发送:

version: '3'
services:
  db:
    image: postgres
    volumes:
      - postgres_data:/var/lib/postgresql/data
    networks:  # <-- connect to the bridge
      - database_network
  redis:
    image: "redis:latest"
    expose:
      - "5672"
    networks:
      - database_network
      - nginx_network
  django:
    build: .
    restart: always
    container_name: tspoptimization
    volumes:
      - .:/opt/services/djangoapp/src
      - static_volume:/opt/services/djangoapp/src/tspweb/static  # <-- bind the static volume
      - media_volume:/opt/services/djangoapp/src/tspweb/media  # <-- bind the media volume
    depends_on:
      - db
      - redis
    networks:
    - nginx_network
    - database_network
  celery:
    build: .
    command: celery -A tspoptimization worker -l info
    volumes:
      - .:/code
    depends_on:
      - db
      - redis
      - django
    links:
      - redis
    networks:
      - nginx_network
      - database_network
  nginx:
    image: nginx:latest
    ports:
     - 8001:80
    volumes:
     - ./config/nginx/conf.d:/etc/nginx/conf.d
     - static_volume:/opt/services/djangoapp/src/tspweb/static  # <-- bind the static volume
     - media_volume:/opt/services/djangoapp/src/tspweb/media  # <-- bind the media volume
    depends_on:
     - django
    networks:
     - nginx_network
networks:
  nginx_network:
    driver: bridge
  database_network:  # <-- add the bridge
    driver: bridge
volumes:
  postgres_data:
  static_volume:
  media_volume:

其实我不知道这是否是应该如何处理的方式，但我不是 devops 的专业人士，所以至少现在这个工作。