apache-spark - 在 Spark 2.4 (Hdinsight) 中使用 Delta Lake 源时出错

Question

低于错误，相同的代码在 Databricks 中有效，但在 Hdinsight 中无效。我还在类路径中添加了 delta 库和 hadoop-azure 库。

io.delta:delta-core_2.11:0.5.0,org.apache.hadoop:hadoop-azure:3.1.3

ERROR ApplicationMaster [Driver]: User class threw exception: com.google.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: com/fasterxml/jackson/module/scala/experimental/ScalaObjectMapper$class
com.google.common.util.concurrent.ExecutionError: java.lang.NoClassDefFoundError: com/fasterxml/jackson/module/scala/experimental/ScalaObjectMapper$class
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2049)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3953)
    at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4873)
    at org.apache.spark.sql.delta.DeltaLog$.apply(DeltaLog.scala:740)
    at org.apache.spark.sql.delta.DeltaLog$.forTable(DeltaLog.scala:712)
    at org.apache.spark.sql.delta.sources.DeltaDataSource.createRelation(DeltaDataSource.scala:169)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:318)
    at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)
    at io.delta.tables.DeltaTable$.forPath(DeltaTable.scala:635)
    

Answer 1

HDInsight打包的jackson-json库版本与spark、deltalake使用的版本冲突
有 2 个选项可以解决这个问题

将 jackson json 2.6.7 版本依赖项打包到您的应用程序中(maven shade 插件或 scala 程序集)

或者

如果您使用的是 jupyter notebook，请设置以下 spark 配置

{"conf":
 {"spark.jars.packages": "io.delta:delta-core_2.11:0.5.0", 
    "spark.driver.extraClassPath":
     "${PATH}/jackson-module-scala_2.11-2.6.7.1.jar;${PATH}/jackson-annotations-2.6.7.jar;
      ${PATH}/jackson-core-2.6.7.jar;
      ${PATH}/jackson-databind-2.6.7.1.jar;
      ${PATH}/jackson-module-paranamer-2.6.7.jar",
   "spark.executor.extraClassPath":
     "${PATH}/jackson-module-scala_2.11-2.6.7.1.jar;${PATH}/jackson-annotations-2.6.7.jar;
      ${PATH}/jackson-core-2.6.7.jar;${PATH}/jackson-databind-2.6.7.1.jar;
      ${PATH}/jackson-module-paranamer-2.6.7.jar",
   "spark.driver.userClassPathFirst":true}}