java - 带有 Spark 的 Kafka 抛出无法初始化类 kafka.utils.Log4jController 错误

Question

我正在尝试使用 Apache Spark 用 java 编写一个 kafka 消费者。由于某些 Log4jController 错误，代码未执行。不知道我错过了什么。

pom.xml文件如下:

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.11</artifactId>
    <version>2.3.0</version>
</dependency>
<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-api</artifactId>
    <version>1.7.25</version>
</dependency>
<dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-streaming_2.11</artifactId>
  <version>2.3.0</version>
  <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-kafka-0-8_2.11</artifactId>
    <version>2.3.0</version>
    <scope>provided</scope>
</dependency>
<dependency>
  <groupId>org.apache.kafka</groupId>
  <artifactId>kafka_2.11</artifactId>
  <version>1.0.0</version>
    <exclusions>
        <exclusion>
            <groupId>org.apache.zookeeper</groupId>
            <artifactId>zookeeper</artifactId>
        </exclusion>
        <exclusion>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
        </exclusion>
    </exclusions>
</dependency>

出现以下错误

5645 [dag-scheduler-event-loop] INFO  org.apache.spark.scheduler.DAGScheduler  - ResultStage 11 (start at RuleEngine.java:431) failed in 0.094 s due to Job aborted due to stage failure: Task 0 in stage 11.0 failed 1 times, most recent failure: Lost task 0.0 in stage 11.0 (TID 8, localhost, executor driver): java.lang.NoClassDefFoundError: Could not initialize class kafka.utils.Log4jController$

编辑:

我能够通过更改 pom.xml 中的 kafka 客户端版本来解决该问题

      <dependency>
    <groupId>org.scala-lang</groupId>
    <artifactId>scala-library</artifactId>
    <version>2.11.0</version>
  </dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.11</artifactId>
    <version>2.3.0</version>
</dependency>
<dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-streaming_2.11</artifactId>
  <version>2.3.0</version>
  <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-streaming-kafka-0-8_2.11</artifactId>
    <version>2.3.0</version>
</dependency>
<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka_2.11</artifactId>
    <version>0.8.2.2</version>
</dependency>

Answer 1

检查你的pom，问题似乎是你使用的是kafka 1.0.0，但使用的是spark-streaming-kafka-0-8，它需要kafka 0.8。 事实上，searching for kafka.utils.Log4jController 显示它是 0.8.1 和 0.8.2 版本中 kafka-clients 库的一部分，但不在更高版本中。我不是 Spark 专家，但我认为您只需要找到与您的 kafka 版本匹配的 Spark-streaming-kafka 库版本。希望有帮助