java - Spark : configuration file 'metrics.properties'

标签 java apache-spark metrics

Spark 的配置文件(用于检索指标),即 /conf/metrics.properties,说明以下内容:

Within an instance, a "source" specifies a particular set of grouped metrics. there are two kinds of sources:

  1. Spark internal sources, like MasterSource, WorkerSource, etc, which will collect a Spark component's internal state. Each instance is paired with a Spark source that is added automatically.

  2. Common sources, like JvmSource, which will collect low level state. These can be added through configuration options and are then loaded using reflection.

下面是一些示例,如

master.source.jvm.class=org.apache.spark.metrics.source.JvmSource

但是,没有示例解释如何从 MasterSourceWorkerSource 等其他来源获取指标,并且仅替换 JvmSource 并不能解决问题。

建议?

最佳答案

MasterSource 或 WorkerSource(均为内部源)会通过指标自动跟踪,因此无需明确指定。

例如,

master.sink.cvs.class=org.apache.spark.metrics.sink.CsvSink

每 10 秒将主实例指标输出到 /tmp 目录。

关于java - Spark : configuration file 'metrics.properties' ,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/31614169/

相关文章:

apache-spark - 如何在Spark 3.0中运行hadoop命令?

java - 如何使用 spring-boot 显示 Dropwizard Metrics Servlet?

java - 如何检查循环java中的条件?

java - 类中的静态泛型方法 - Java

scala - Spark 2.2.0 - 如何将 DataFrame 写入/读取 DynamoDB

android - 计算丢包、抖动和延迟

python - 如何在Python中找到真阳性、真阴性、假阳性、假阴性

JAVA/Swing(Eclipse): Program works step by step but not in normal execution

java - 为什么 Apache poi 的 Workbook.close() 方法将内容写入输入文件?

apache-spark - Spark 2.0读取本地parquet文件