为了避免消费者崩溃并恢复时出现冗余消息，我禁用了偏移量的自动提交并手动提交它们。
现在的问题是，如果不同机器上的消费者进程访问同一主题，他们会收到唯一的消息吗？从理论上讲，手动提交会导致不同机器上收到冗余消息。
在我的本地计算机上，我运行了两个订阅同一主题的 java 消费者实例，并且它们收到了重复的消息。这件事该怎么处理呢？我正在使用高级消费者
最佳答案
由于 Kafka 的消息传递语义是至少一次，因此您应该实现自己的代码来保证 Kafka 中的精确一次语义。
- 最多一次:消息可能会丢失，但永远不会重新传送。
- 至少一次:消息永远不会丢失，但可以重新传送。
- 恰好一次:这是人们真正想要的，每条消息都会传递一次且仅传递一次。
来自4.6 Message Delivery Semantics在卡夫卡文档中:
So what about exactly once semantics (i.e. the thing you actually want)? The limitation here is not actually a feature of the messaging system but rather the need to co-ordinate the consumer's position with what is actually stored as output. The classic way of achieving this would be to introduce a two-phase commit between the storage for the consumer position and the storage of the consumers output. But this can be handled more simply and generally by simply letting the consumer store its offset in the same place as its output. This is better because many of the output systems a consumer might want to write to will not support a two-phase commit. As an example of this, our Hadoop ETL that populates data in HDFS stores its offsets in HDFS with the data it reads so that it is guaranteed that either data and offsets are both updated or neither is. We follow similar patterns for many other data systems which require these stronger semantics and for which the messages do not have a primary key to allow for deduplication.
Kafka FAQ 中还有一个问题有类似的答案:How do I get exactly-once messaging from Kafka?
关于java - 运行在不同机器上的Kafka消费者组可以接收到唯一的消息吗？，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/29514245/