我正在使用 Kinesis 设置 Spark Streaming 项目,当我尝试连接到 Kinesis 流时,我从 Spark 收到以下错误:
ERROR ShardSyncTask: Caught exception while sync'ing Kinesis shards and leases
com.amazonaws.services.kinesis.clientlibrary.exceptions.internal.KinesisClientLibIOException: Parent shard shardId-000000000000 exists but not the child shard shardId-000000000002
当我将测试数据发布到此流或使用基本 Amazon 库从流中读取数据时,我没有收到任何错误,这仅在我尝试与 Spark 连接时发生。
以下是我用于测试的代码:
val conf = new SparkConf().setMaster("local[2]").setAppName("KinesisCounter")
val ssc = new StreamingContext(conf, Seconds(1))
val rawStream = KinesisUtils.createStream(ssc, "dev-test", "kinesis.us-east-1.amazonaws.com", Duration(1000), InitialPositionInStream.TRIM_HORIZON, StorageLevel.MEMORY_ONLY)
rawStream.map(msg => new String(msg)).count.print
最佳答案
您在 Kinesis 上有多少个分片?
我会做的是:
Changing the application name or stream name can lead to Kinesis errors in some cases. If you see errors, you may need to manually delete the DynamoDB table
希望能帮助到你。
关于spark-streaming - 父分片存在但不存在子分片,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/30199338/