apache-kafka - 卡夫卡 - 经纪人 : Message size too large

Question

我收到 Message size too large异常(exception)，当我尝试发送超过 1 Mb 大小的消息时。当我尝试生成消息时，错误出现在我的客户端应用程序中。经过一番谷歌搜索后，我发现应该更改设置以增加最大消息大小。好吧，我是在 /kafka/config/server.properties 中做到的文件。我添加了接下来的 2 个设置:

message.max.bytes=15728640
replica.fetch.max.bytes=15728640

另外，我添加了 fetch.message.max.bytes=15728640到 /kafka/config/consumer.properties文件。所有其他设置保持默认。

我重新启动了 kafka 服务器，但仍然出现相同的错误。

P.S Kafka 版本是 1.1.0。

Answer 1

您有正确的配置，但是您还需要设置 最大请求大小 在生产者方面。

props.put(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, 15728640);

max.request.size The maximum size of a request in bytes. This setting will limit the number of record batches the producer will send in a single request to avoid sending huge requests. This is also effectively a cap on the maximum record batch size.

在 Broker 方面，您已经配置了以下应该可以工作的参数

message.max.bytes The largest record batch size allowed by Kafka.

replica.fetch.max.bytes The number of bytes of messages to attempt to fetch for each partition. This is not an absolute maximum if the first record batch in the first non-empty partition of the fetch is larger than this value, the record batch will still be returned to ensure that progress can be made. The maximum record batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config).

在主题端 max.message.bytes 如果您已经在代理端设置了 message.max.bytes，则不需要

max.message.bytes - this is the largest size of the message the broker will allow being appended to the topic. This size is validated pre-compression. (Defaults to broker's message.max.bytes.)

引用
https://kafka.apache.org/documentation/