amazon-kinesis - Kinesis 分片读取限制为 2 mib，那么获取记录调用怎么可能达到 10mib

Question

我指的是这个document其中显示“每个分片通过 GetRecords 最多可支持每秒 2 MiB 的最大总数据读取速率。如果对 GetRecords 的调用返回 10 MiB，则接下来 5 秒内进行的后续调用将引发异常。” 我试图理解 getRecords 调用怎么可能 get(10Mib) 超过 2mib 的分片限制？碎片达到 2 mib 限制后不会停止/抛出错误吗？

提前致谢

Answer 1

这句话看起来自相矛盾。他们应该重新措辞。

您应该考虑文档中的前两个陈述来理解上下文。

摘自上述文档，

GetRecords can retrieve up to 10 MiB of data per call from a single shard, and up to 10,000 records per call. Each call to GetRecords is counted as one read transaction.

Each shard can support up to five read transactions per second. Each read transaction can provide up to 10,000 records with an upper limit of 10 MiB per transaction.

Each shard can support up to a maximum total data read rate of 2 MiB per second via GetRecords. If a call to GetRecords returns 10 MiB, subsequent calls made within the next 5 seconds throw an exception.

根据我使用 Kinesis 的经验，它们的实际含义是，每个分片对于 GetRecords 调用的读取速率限制为每秒 2 MiB，并且此速率限制是在 GetRecords 调用开始时超过一秒的窗口中计算的。

我不确定 Kinesis 内部实现，但我知道 internals of Kafka 。在 Kafka 中，分区(与 Kinesis 中的分片相同)进一步分为段，这些段基本上是日志文件。因此，每条消息都作为一个条目存储在日志文件中。

我怀疑他们已经通过以下方式实现了 GetRecords 服务器端 API，

Python 伪代码:

current_timestamp = datetime.now
seconds_diff = (LAST_SUCCESSFUL_CALL.timestamp - current_timestamp).total_seconds()
if LAST_SUCCESSFUL_CALL.data_size > (seconds_diff * 2 Mib):
  LAST_SUCCESSFUL_CALL.data_size = LAST_SUCCESSFUL_CALL.data_size - (seconds_diff * 2 Mib)
  throw Error
else
  records = data_store.find_next_records_from_segments(10 MiB)
  # Here, implementation does not limit the records because sequential disk reading is always faster. 
  # So, It will be better to get as much records it has with some upper cap of 10 MiB or till the end of segment. 
  LAST_SUCCESSFUL_CALL.data_size = records.data_size
  LAST_SUCCESSFUL_CALL.timestamp = current_timestamp
  return records

通过将速率限制检查分散到之前的调用中，他们使实现变得更加简单。

它也最适合消费者可以快速追上记录的流处理应用程序。

例如，假设发生以下事件

T1 -> Ingest 1 MiB in shard, Consumer is busy on processing fetched data, Pending data = 1 MiB
T2 -> Ingest 1 MiB in shard, Consumer is busy on processing fetched data, Pending data = 2 MiB
T3 -> Ingest 1 MiB in shard, Consumer is busy on processing fetched data, Pending data = 3 MiB
T4 -> Ingest 1 MiB in shard, Consumer is busy on processing fetched data, Pending data = 4 MiB
T5 -> Ingest 1 MiB in shard, Consumer is busy on processing fetched data, Pending data = 5 MiB
T6 -> Ingest 1 MiB in shard, Consumer becomes idle and does GetRecords, gets 5 MiB data, Pending data = 1 MiB
T7 -> No new data ingestion, Consumer is busy on processing fetched data 
T8 -> No new data ingestion, Consumer is busy on processing fetched data 
T9 -> Consumer becomes Idle and does GetRecords, gets 1 MiB data. Pending data = 0 MiB 

因此，T7 到 T8，消费者使用 2 秒来完全处理 5 MiB 数据，而不是分别为每个 2 MiB 数据创建 GetRecords。在这里，我们保存网络调用和磁盘搜索。

总而言之，

Will not the shard stop/throw erro after it reached the 2 mib limit?

不，不会。但是在接下来的几秒中进行的 GetRecords 将会抛出错误。但大多数情况下，您的消费者将在接下来的几秒钟内处理您在第一次 GetRecords 调用时收到的 10 MiB 数据，而不是查询新数据。所以，您不必太担心。