java - 使用 Spark 连接 MariaDB 时出现 ClassNotFoundException

Question

我正在尝试将 Spark 连接到 MariaDB，但出现了

Exception in thread "main" java.lang.ClassNotFoundException: org.mariadb.jdbc.Driver

现在，在将其标记为重复之前，我知道该错误意味着什么，并且我已经阅读了不同的线程，但没有成功。

这是我的build.sbt

scalaVersion := "2.11.8"

name := "SparkJdbc"

libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-core" % "2.0.2",
    "org.apache.spark" %% "spark-sql" % "2.0.2",
    "org.mariadb.jdbc" % "mariadb-java-client" % "1.5.2"
)

我也有单独的 jar ，并且我已经尝试过

spark-submit --class SparkJdbc target/scala-2.11/sparkjdbc_2.11-0.1-SNAPSHOT.jar 
--jars /path/to/mariadb-java-client-1.5.2.jar

但这也因相同的错误而失败。

以下是我用来连接的代码

val (driver, url, username, password) = ("org.mariadb.jdbc.Driver", "jdbc:mysql://localhost/db1", "user", "password");

Class.forName(driver).newInstance;

val data = spark.sqlContext.read
    .format("jdbc")
    .option("url", url)
    .option("user", username)
    .option("password", password)
    .option("dbtable", "SELECT * FROM db1.football LIMIT 10")
    .load();

Answer 1

我可以看到两个可能的错误:

1. --jars 应在 jar 名称之前使用。来自 documentation ，关于 Jar 名称后面的参数:

Arguments passed to the main method of your main class, if any

文件应位于所有节点上的同一路径上或位于 HDFS 等分布式文件系统中 - 如果您仅在主节点或驱动程序中获取文件，则应用程序将引发错误

编辑:

你should还设置 --driver-class-path=/path/to/jar/with/driver.jar

取自文档的其他选项是:

The JDBC driver class must be visible to the primordial class loader on the client session and on all executors. This is because Java’s DriverManager class does a security check that results in it ignoring all drivers not visible to the primordial class loader when one goes to open a connection. One convenient way to do this is to modify compute_classpath.sh on all worker nodes to include your driver JARs.