docker - 设置太阳鸟遥测Kafka DRUID和超集

Question

我正在尝试根据移动事件创建分析仪表板。我想将所有组件都 docker 化到Docker中的容器中，并将其部署在localhost中并创建一个分析仪表板。

Sunbird遥测https://github.com/project-sunbird/sunbird-telemetry-service

Kafka https://github.com/wurstmeister/kafka-docker

德鲁伊https://github.com/apache/incubator-druid/tree/master/distribution/docker

super 集https://github.com/apache/incubator-superset

我做了什么

德鲁伊

我执行了命令docker build -t apache/incubator-druid:tag -f distribution/docker/Dockerfile .

我执行了命令docker-compose -f distribution/docker/docker-compose.yml up

一切执行完毕后，打开http://localhost:4008/并查看DRUID运行

完成构建和运行需要 3.5小时

卡夫卡

导航到kafka文件夹

docker-compose up -d执行了此命令

问题
当我们执行druid时，zookeeper将开始运行，而当我们启动kafka时，docker文件将启动另一个zookeeper，而我无法在kafka和zookeeper之间建立连接。
在我开始sunbird遥测并尝试创建主题并从sunbird连接kafka之后，它没有连接。

我不明白我在做什么错。
我们可以告诉kafka共享由DRUID 启动的zookeeper。我是这个环境和这些堆栈的新手。
我正在研究这些堆栈。难道我做错了什么。谁能指出如何通过docker正确连接kafka和druid。

注意:-我正在Mac上运行所有这些

我的Kafka撰写文件

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    build: .
    ports:
      - "9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: **localhost ip**
      KAFKA_ZOOKEEPER_CONNECT: **localhost ip**:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

Answer 1

Can we tell kafka to share the zookeeper started by DRUID

您会将所有服务放在同一个撰写文件中。

德鲁伊卡夫卡连接在这里列出

https://github.com/apache/incubator-druid/blob/master/distribution/docker/environment#L31

您可以将KAFKA_ZOOKEEPER_CONNECT设置为相同的地址，可以

例如，下载上面的文件并将Kafka添加到Druid Compose文件中...

version: "2.2"

volumes:
  metadata_data: {}
  middle_var: {}
  historical_var: {}
  broker_var: {}
  coordinator_var: {}
  overlord_var: {}
  router_var: {}

services:

  # TODO: Add sunbird

  postgres:
    container_name: postgres
    image: postgres:latest
    volumes:
      - metadata_data:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=FoolishPassword
      - POSTGRES_USER=druid
      - POSTGRES_DB=druid

  # Need 3.5 or later for container nodes
  zookeeper:
    container_name: zookeeper
    image: zookeeper:3.5
    environment:
      - ZOO_MY_ID=1

  druid-coordinator:
    image: apache/incubator-druid
    container_name: druid-coordinator
    volumes:
      - coordinator_var:/opt/druid/var
    depends_on: 
      - zookeeper
      - postgres
    ports:
      - "3001:8081"
    command:
      - coordinator
    env_file:
      - environment

  # renamed to druid-broker
  druid-broker: 
    image: apache/incubator-druid
    container_name: druid-broker
    volumes:
      - broker_var:/opt/druid/var
    depends_on: 
      - zookeeper
      - postgres
      - druid-coordinator
    ports:
      - "3002:8082"
    command:
      - broker
    env_file:
      - environment

 # TODO: add other Druid services

 kafka:
    image: wurstmeister/kafka
    ports:
      - "9092"
    depends_on: 
      - zookeeper
    environment:
      KAFKA_ADVERTISED_HOST_NAME: kafka
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181/kafka  # This is the same service that Druid is using