我正在Windows的Docker中运行,这是我的NiFi设置:
PublishKafka处理器的详细信息:
ConsumeKafka处理器的详细信息:
这是我的docker-compose文件(注意:192.168.1.50是我的静态内部主机IP):
version: '3'
services:
Jenkins:
container_name: Jenkins
restart: on-failure
depends_on:
- NiFi
image: jenkins:latest
ports:
- "32779:50000"
- "32780:8080"
NiFi:
container_name: NiFi
image: xemuliam/nifi:latest
restart: on-failure
depends_on:
- kafka
ports:
- "32784:8089"
- "32783:8080"
- "32782:8081"
- "32781:8443"
labels:
com.foo: myLabel
zookeeper:
container_name: Zookeeper
image: wurstmeister/zookeeper
restart: on-failure
#network_mode: host
ports:
- "2181:2181"
kafka:
#container_name: Kafka
image: wurstmeister/kafka
depends_on:
- zookeeper
#restart: on-failure
#network_mode: host
ports:
- "9092"
environment:
#KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.1.50:9092
#KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
KAFKA_CREATE_TOPICS: "MainIngestionTopic:1:1"
KAFKA_ZOOKEEPER_CONNECT: 192.168.1.50:2181
KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://192.168.1.50:9094
KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
volumes:
- ./var/run/docker.sock:/var/run/docker.sock
当我拖尾Kafka容器日志时,可以看到我的主题是通过docker-compose成功创建的。
邮件已成功传递到NiFi中的PublishKafka处理器,但随后无法发布。订阅相同主题的ConsumeKafka处理器永远不会收到该消息。
NiFi容器日志显示以下内容:
2018-05-28 19:46:18,792 ERROR [Timer-Driven Process Thread-1] o.a.n.p.kafka.pubsub.PublishKafka PublishKafka[id=b2503f49-acc9-38f5-86f9-5029e2768b68] Failed to send all message for StandardFlowFileRecord[uuid=b3f6f818-34d3-42a9-9d6e-636cf17eb138,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1527533792820-1, container=default, section=1], offset=5, length=5],offset=0,name=8151630985100,size=5] to Kafka; routing to failure due to org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 5000 ms.: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 5000 ms.
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 5000 ms.
2018-05-28 19:46:18,792 INFO [Timer-Driven Process Thread-1] o.a.kafka.clients.producer.KafkaProducer Closing the Kafka producer with timeoutMillis = 5000 ms.
我尝试从Kafka容器本身内部发布到该主题,但是也失败了:
我已经整理了文档并阅读了许多线程来尝试解决此问题,但这仍然是一个问题。任何帮助将不胜感激!
最佳答案
您不能在NiFi的“Kafka Brokers”属性中使用localhost,除非该代理实际在与NiFi运行的主机相同的主机上运行。由于每个服务都在docker容器中,因此kafka的容器必须具有可以使用的特定主机名或ip。
关于docker - 无法从NiFi发送到Kafka,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/50572797/