apache-spark - 在 Spark Streaming 中禁用 AWS Kinesis 的 CloudWatch

Question

我想知道是否可以？

这里是代码:numStreams 我通过使用 AmazonKinesisClient API 获取它

 // Create the Kinesis DStreams
    List<JavaDStream<byte[]>> streamsList = new ArrayList<>(numStreams);
    for (int i = 0; i < numStreams; i++) {
      streamsList.add(
              KinesisUtils.createStream(jssc, kinesisAppName, streamName, endpointUrl, regionName,
              InitialPositionInStream.TRIM_HORIZON, kinesisCheckpointInterval,
              StorageLevel.MEMORY_AND_DISK_2(),accessesKey,secretKey)
      );
    }

我尝试查看 API，但找不到任何有关禁用 Apache Streaming CloudWatch 的引用。

这是我尝试摆脱的警告:

17/01/23 17:46:29 WARN CWPublisherRunnable: Could not publish 16 datums to CloudWatch com.amazonaws.AmazonServiceException: User: arn:aws:iam:::user/Kinesis_Service is not authorized to perform: cloudwatch:PutMetricData (Service: AmazonCloudWatch; Status Code: 403; Error Code: AccessDenied; Request ID: *****) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1377) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:923) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:701) at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:453) at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:415) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:364) at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.doInvoke(AmazonCloudWatchClient.java:984) at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke(AmazonCloudWatchClient.java:954) at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.putMetricData(AmazonCloudWatchClient.java:853) at com.amazonaws.services.kinesis.metrics.impl.DefaultCWMetricsPublisher.publishMetrics(DefaultCWMetricsPublisher.java:63) at com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable.runOnce(CWPublisherRunnable.java:144) at com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable.run(CWPublisherRunnable.java:90) at java.lang.Thread.run(Unknown Source)

Answer 1

前言:我知道这是一个老问题，但刚刚遇到这个问题，因此为遇到 Spark <= 2.3.3 问题的任何人发布解决方案

构建客户端时，可以使用 withMetrics 方法在 KCL(Kinesis 客户端)库级别禁用 Cloudwatch 指标报告。

不幸的是，Spark KinesisInputDStream 方法没有公开更改此设置的方法，更糟糕的是，默认级别是“DETAILED”，每 10 秒发送 10 秒的指标。

我采取的禁用它的方法是向 KinesisInputDStream 中的方法 cloudWatchCredentials 提供无效凭证。 IE:.cloudWatchCredentials(SparkAWSCredentials.builder.basicCredentials("DISABLED", "DISABLED").build())

然后是每次更新时 CloudWatchAsyncClient 日志记录警告的问题，我通过在 Spark log4j.properties 配置中设置以下内容来禁用该警告:

# Set Kinesis logging metrics to Warn - Since we intentionally provide
# wrong credentials in order to disable cloudwatch logging. Bad credential
# warning are logged at WARN level - so we still get errors.
log4j.logger.com.amazonaws.services.kinesis.metrics=ERROR

这将仅抑制对指标包类的警告(例如您提到的那个)，但不会抑制错误，以防需要这些错误。

这远非理想的解决方案，但这允许我们在部署现有 Spark 版本的同时部署解决方案。

后续步骤:向 Spark 开具票证，以便他们有望允许我们在下一版本中禁用它。

编辑 - 创建:https://issues.apache.org/jira/browse/SPARK-27420用于跟踪