docker - 使用docker将kafka主题数据写入redis

Question

我正在使用this repository kafka连接到redis。

说明:我想做的是使用docker将kafka主题数据写入redis。他们创建了一个自述文件来指导如何设置 kafka 的配置:

curl -s -X POST -H 'Content-Type: application/json' --data @connector.json http://localhost:8083/connectors

connector.json 文件包含:

{
  "config" : {
    "name" : "RedisSinkConnector1",
    "connector.class" : "com.github.jcustenborder.kafka.connect.redis.RedisSinkConnector",
    "tasks.max" : "1",
    "topics" : "mostafa"
  }
}

问题:我知道如何在kafka中创建新主题，但问题是我不知道如何更改docker-compose或测试连接。虽然我在kafka中创建了一个新主题，但redis数据库中没有显示任何内容!

如果有人能帮助我，我将不胜感激。

Answer 1

以下配置可以解决该问题。

{
  "name" : "RedisSinkConnector1",
  "config" : {
    "connector.class" : "com.github.jcustenborder.kafka.connect.redis.RedisSinkConnector",
    "tasks.max" : "1",
    "topics" : "mostafa",
    "redis.hosts": "redis:6379",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "value.converter": "org.apache.kafka.connect.storage.StringConverter"
  }
}

在 docker-compose 文件中添加 kafka-connect

kafka-connect:
    hostname: kafka-connect
    image: confluentinc/cp-kafka-connect:latest
    container_name: kafka-connect
    ports:
      - 8083:8083
    depends_on:
      - schema-registry
      **- redis**
    environment:
      CONNECT_BOOTSTRAP_SERVERS: kafka:9092
      CONNECT_REST_PORT: 8083
      CONNECT_GROUP_ID: "quickstart-avro"
      CONNECT_CONFIG_STORAGE_TOPIC: "quickstart-avro-config"
      CONNECT_OFFSET_STORAGE_TOPIC: "quickstart-avro-offsets"
      CONNECT_STATUS_STORAGE_TOPIC: "quickstart-avro-status"
      CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
      CONNECT_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
      CONNECT_VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
      CONNECT_INTERNAL_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
      CONNECT_INTERNAL_VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
      **CONNECT_REST_ADVERTISED_HOST_NAME: "kafka-connect"**
      CONNECT_LOG4J_ROOT_LOGLEVEL: DEBUG
      CONNECT_PLUGIN_PATH: "/usr/share/java,/etc/kafka-connect/jars"
    volumes:
      - $PWD/jars:/etc/kafka-connect/jars

depends_on redis 和 CONNECT_REST_ADVERTISED_HOST_NAME 变量对于解决此问题非常重要