mysql - Spark 未找到类为 com.mysql.jdbc.Driver 的已注册驱动程序

Question

我正在使用 CDH 5.7.0 和 PySpark。当我运行诸如 RDD.count() 之类的操作时，它显示错误:没有找到带有类 com.mysql.jdbc.Driver 的注册驱动程序

步骤如下

pyspark --driver-class-path/usr/share/java/mysql-connector-java.jar (每个节点上的/usr/share/java/mysql-connector-java.jar)

>>>url ="jdbc:mysql://host/spark?user=root&password=test"
>>> stock_data=sqlContext.read.format("jdbc").option("url",url).option("dbtable","StockPrices").load()
>>> stock_data.printSchema()
root
 |-- date: string (nullable = true)
 |-- open: double (nullable = true)
 |-- high: double (nullable = true)
 |-- low: double (nullable = true)
 |-- close: double (nullable = true)
 |-- volume: long (nullable = true)
 |-- adjclose: double (nullable = true)
>>> stock_data.count()
......
    at java.lang.reflect.Method.invoke(Method.java:606)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381)
    at py4j.Gateway.invoke(Gateway.java:259)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:209)
    at java.lang.Thread.run(Thread.java:745)
**Caused by: java.lang.IllegalStateException: Did not find registered driver with class com.mysql.jdbc.Driver**
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$2$$anonfun$3.apply(JdbcUtils.scala:58)
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$2$$anonfun$3.apply(JdbcUtils.scala:58)
    at scala.Option.getOrElse(Option.scala:120)
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$2.apply(JdbcUtils.scala:57)
    at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$createConnectionFactory$2.apply(JdbcUtils.scala:52)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$$anon$1.<init>(JDBCRDD.scala:347)
    at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD.compute(JDBCRDD.scala:339)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)

Answer 1

终于找到了...你需要在conf下，有一个default.conf文件，添加spark.executor.extraClassPath mysql.jar，这样executor就可以找到驱动了。