symfony - 将带有Filebeat的symfony日志文件传输到本地docker-环境中的graylog

Question

描述

我正在尝试在本地docker环境中像在生产系统上一样构建相同的配置。花了一些时间调查并重建docker容器设置后，仍然无法使它正常工作，并且Graylog没有收到任何数据。

概述和中期结果

基于symfony的应用程序使用

Web，php和db容器

symfony在php-container中的localhost上正常运行，并生成日志文件

symfony-log文件位于此处:/var/www/html/var/logs/*.log

symfony-logfiles格式为json / gelf

启动完整的组合

时，所有其他容器也已启动并运行

filebeat配置基于

下面的第一个链接

filebeat.yml似乎检索在任何容器

中找到的任何日志文件

filebeat配置为直接将数据传输到elasticsearch

elasticsearch将数据保留在mongodb中

持久存储在docker

中命名卷中的所有与Graylog相关的数据

另外，我正在Mac上使用docker-sync

docker-compose.yml基于以下资源:

https://github.com/jochenchrist/docker-logging-elasticsearch

http://docs.graylog.org/en/2.4/pages/installation/docker.html?highlight=docker

https://www.elastic.co/guide/en/beats/filebeat/6.3/running-on-docker.html

https://www.elastic.co/guide/en/beats/filebeat/6.3/filebeat-reference-yml.html

配置文件

# Monolog Configuration
monolog:
  channels: [graylog]
  handlers:
    graylog:
      type:      stream
      formatter: line_formatter
      path:      "%kernel.logs_dir%/graylog.log"
      channels:  [graylog]

docker-compose.yml

version: "3"
services:
    web:
        image: nginx
        ports:
            - "80:80"
            - "443:443"
        links:
            - php
        volumes:
            - ./docker-config/nginx.conf:/etc/nginx/conf.d/default.conf
            - project-app-sync:/var/www/html
            - ./docker-config/localhost.crt:/etc/nginx/ssl/localhost.crt
            - ./docker-config/localhost.key:/etc/nginx/ssl/localhost.key

    php:
        build:
            context: .
            dockerfile: ./docker-config/Dockerfile-php
        links:
            - graylog
        volumes:
            - project-app-sync:/var/www/html
            - ./docker-config/php.ini:/usr/local/etc/php/php.ini
            - ./docker-config/www.conf:/usr/local/etc/php-fpm.d/www.conf

    db:
        image: mysql
        ports:
            - "3306:3306"
        environment:
            - MYSQL_ALLOW_EMPTY_PASSWORD=yes
            - MYSQL_DATABASE=project
            - MYSQL_USER=project
            - MYSQL_PASSWORD=password
        volumes:
            - ./docker-config/mysql.cnf:/etc/mysql/conf.d/mysql.cnf
            - project-mysql-sync:/var/lib/mysql

    # Graylog / Filebeat

    filebeat:
        build: ./docker-config/filebeat
        volumes:
          - /var/lib/docker/containers:/var/lib/docker/containers:ro
          - /var/run/docker.sock:/var/run/docker.sock
        networks:
          - graylog-network
        depends_on:
          - graylog-elasticsearch

    graylog:
        image: graylog/graylog:2.4
        volumes:
          - graylog-journal:/usr/share/graylog/data/journal
        networks:
          - graylog-network
        environment:
          - GRAYLOG_PASSWORD_SECRET=somepasswordpepper
          - GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
          - GRAYLOG_WEB_ENDPOINT_URI=http://127.0.0.1:9000/api
        links:
          - graylog-mongo:mongo
          - graylog-elasticsearch:elasticsearch
        depends_on:
          - graylog-mongo
          - graylog-elasticsearch
        ports:
          # Graylog web interface and REST API
          - 9000:9000

    graylog-mongo:
        image: mongo:3
        volumes:
            - graylog-mongo-data:/data/db
        networks:
            - graylog-network

    graylog-elasticsearch:
        image: docker.elastic.co/elasticsearch/elasticsearch:5.6.10
        ports:
            - "9200:9200"
        volumes:
            - graylog-elasticsearch-data:/usr/share/elasticsearch/data
        networks:
            - graylog-network
        environment:
            - cluster.name=graylog
            - "discovery.zen.minimum_master_nodes=1"
            - "discovery.type=single-node"
            - http.host=0.0.0.0
            - transport.host=localhost
            - network.host=0.0.0.0
            # Disable X-Pack security: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/security-settings.html#general-security-settings
            - xpack.security.enabled=false
            - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
        ulimits:
            memlock:
                soft: -1
                hard: -1

volumes:
    project-app-sync:
        external: true
    project-mysql-sync: ~
    graylog-mongo-data:
        driver: local
    graylog-elasticsearch-data:
        driver: local
    graylog-journal:
        driver: local

networks:
    graylog-network: ~

Filebeat容器的Dockerfile

FROM docker.elastic.co/beats/filebeat:6.3.1
COPY filebeat.yml /usr/share/filebeat/filebeat.yml
# must run as root to access /var/lib/docker and /var/run/docker.sock
USER root
RUN chown root /usr/share/filebeat/filebeat.yml
# dont run with -e, to disable output to stderr
CMD [""]

filebeat.yml

filebeat.prospectors:
- type: docker
  paths:
    - '/var/lib/docker/containers/*/*.log'
    # path to symfony based logs
    - '/var/www/html/var/logs/*.log'
  containers.ids: '*'

processors:
  - decode_json_fields:
      fields: ["host","application","short_message"]
      target: ""
      overwrite_keys: true
  - add_docker_metadata: ~

output.elasticsearch:
  # transfer data to elasticsearch container?
  hosts: ["localhost:9200"]

logging.to_files: true
logging.to_syslog: false

Graylog后端

设置完此docker组合后，我启动了Graylog网络 View ，并按如下所述设置了收集器和输入:

http://docs.graylog.org/en/2.4/pages/collector_sidecar.html#step-by-step-guide

也许我完全误解了它是如何工作的。我不太确定Elastic的Beats是否与filebeats容器相同，以及sidecar收集器是否是我忘记添加的东西。也许我没有正确配置收集器并在graylog中输入了？

我将根据我的问题感谢任何帮助或工作示例...

Answer 1

Graylog似乎正在容器中的http://127.0.0.1:9000/api上运行。您可能希望将其作为http://graylog:9000/api或http://0.0.0.0:9000/api运行

从其他任何图像中访问其他图像的操作将与service文件中定义的docker-compose.yml名称相同。 graylog-elasticsearch的URL类似于:http://graylog-elasticsearch/....如果您要发布到localhost，它将保留在其自己的图像中。

希望这会帮助您找到解决方案。