apache-kafka - Kafka 单个消费者在一个组中失败

Question

我正处于探索 Kafka 0.8.1.1 版的初始阶段。

我已经成功运行了具有多个分区的消费者组示例，并且它在消费者之间很好地分发了消息。

我想运行的一个测试用例是当组中的一个消费者突然死亡时(例如， kill -9 )。当我这样做时，我预计会发生重新平衡，但它没有发生。那么，我可以做这些事情之一吗？

使用 API 触发重新平衡

将 kafka 配置为等待消费者事件的特定时间，并在假设它不正常关闭时自动重新平衡。

这里的问题是，分配给死消费者的分区中的所有消息都保留在队列中，并且在重新平衡发生之前永远不会被处理。

Answer 1

重新平衡将自动发生，可以在消费者配置 (zookeeper.session.timeout.ms) 中设置。根据文档

zookeeper.session.timeout.ms : ZooKeeper session timeout. If the consumer fails to heartbeat to zookeeper for this period of time it is considered dead and a rebalance will occur. default value is 6000 ms

同一组中的另一个实时消费者将在超时间隔后开始接收消息。
根据您的要求配置此超时值。
还有一些来自 kafka 文档的更多信息:

Consumer rebalancing fails (you will see ConsumerRebalanceFailedException): This is due to conflicts when two consumers are trying to own the same topic partition. The log will show you what caused the conflict (search for "conflict in ").

1. If your consumer subscribes to many topics and your ZK server is busy, this could be caused by consumers not having enough time to see a consistent view of all consumers in the same group. If this is the case, try Increasing rebalance.max.retries and rebalance.backoff.ms.
2. Another reason could be that one of the consumers is hard killed. Other consumers during rebalancing won't realize that consumer is gone after zookeeper.session.timeout.ms time. In the case, make sure that rebalance.max.retries * rebalance.backoff.ms > zookeeper.session.timeout.ms.