mysql - 如何包含 MySQL 连接器 jar

标签 mysql apache-spark

tong@tong-VirtualBox:/usr/local/spark$ bin/sparkling-shell --jars /home/tong/sparkling-water/mysql-connector-java-5.1.36-bin.jar
bash: bin/sparkling-shell: No such file or directory
tong@tong-VirtualBox:/usr/local/spark$ bin/spark-shell --jars /home/tong/sparkling-water/mysql-connector-java-5.1.36-bin.jar
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/09/17 11:19:27 INFO SecurityManager: Changing view acls to: tong
15/09/17 11:19:27 INFO SecurityManager: Changing modify acls to: tong
15/09/17 11:19:27 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(tong); users with modify permissions: Set(tong)
15/09/17 11:19:29 INFO HttpServer: Starting HTTP Server
15/09/17 11:19:29 INFO Utils: Successfully started service 'HTTP class server' on port 43333.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.4.1
      /_/

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_80)
Type in expressions to have them evaluated.
Type :help for more information.
15/09/17 11:19:43 WARN Utils: Your hostname, tong-VirtualBox resolves to a loopback address: 127.0.1.1; using 10.23.36.82 instead (on interface eth0)
15/09/17 11:19:43 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
15/09/17 11:19:43 INFO SparkContext: Running Spark version 1.4.1
15/09/17 11:19:43 INFO SecurityManager: Changing view acls to: tong
15/09/17 11:19:43 INFO SecurityManager: Changing modify acls to: tong
15/09/17 11:19:43 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(tong); users with modify permissions: Set(tong)
15/09/17 11:19:45 INFO Slf4jLogger: Slf4jLogger started
15/09/17 11:19:45 INFO Remoting: Starting remoting
15/09/17 11:19:46 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@10.23.36.82:35469]
15/09/17 11:19:46 INFO Utils: Successfully started service 'sparkDriver' on port 35469.
15/09/17 11:19:46 INFO SparkEnv: Registering MapOutputTracker
15/09/17 11:19:47 INFO SparkEnv: Registering BlockManagerMaster
15/09/17 11:19:47 INFO DiskBlockManager: Created local directory at /tmp/spark-f8f4de26-e607-416f-9fed-f37440bd3878/blockmgr-ed450a0c-5719-4721-98b7-fd6e4664a7d4
15/09/17 11:19:47 INFO MemoryStore: MemoryStore started with capacity 267.3 MB
15/09/17 11:19:47 INFO HttpFileServer: HTTP File server directory is /tmp/spark-f8f4de26-e607-416f-9fed-f37440bd3878/httpd-99f7d1e3-e6d8-4a06-8ae0-65d0fb76a038
15/09/17 11:19:47 INFO HttpServer: Starting HTTP Server
15/09/17 11:19:47 INFO Utils: Successfully started service 'HTTP file server' on port 51511.
15/09/17 11:19:47 INFO SparkEnv: Registering OutputCommitCoordinator
15/09/17 11:19:48 INFO Utils: Successfully started service 'SparkUI' on port 4040.
15/09/17 11:19:48 INFO SparkUI: Started SparkUI at http://10.23.36.82:4040
15/09/17 11:19:48 INFO SparkContext: Added JAR file:/home/tong/sparkling-water/mysql-connector-java-5.1.36-bin.jar at http://10.23.36.82:51511/jars/mysql-connector-java-5.1.36-bin.jar with timestamp 1442506788779
15/09/17 11:19:49 INFO Executor: Starting executor ID driver on host localhost
15/09/17 11:19:49 INFO Executor: Using REPL class URI: http://10.23.36.82:43333
15/09/17 11:19:50 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 34826.
15/09/17 11:19:50 INFO NettyBlockTransferService: Server created on 34826
15/09/17 11:19:50 INFO BlockManagerMaster: Trying to register BlockManager
15/09/17 11:19:50 INFO BlockManagerMasterEndpoint: Registering block manager localhost:34826 with 267.3 MB RAM, BlockManagerId(driver, localhost, 34826)
15/09/17 11:19:50 INFO BlockManagerMaster: Registered BlockManager
15/09/17 11:19:51 INFO SparkILoop: Created spark context..
Spark context available as sc.
15/09/17 11:19:54 INFO HiveContext: Initializing execution hive, version 0.13.1
15/09/17 11:19:55 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
15/09/17 11:19:55 INFO ObjectStore: ObjectStore, initialize called
15/09/17 11:19:56 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
15/09/17 11:19:56 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
15/09/17 11:19:56 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
15/09/17 11:19:57 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
15/09/17 11:20:04 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
15/09/17 11:20:04 INFO MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5.  Encountered: "@" (64), after : "".
15/09/17 11:20:06 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
15/09/17 11:20:06 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
15/09/17 11:20:13 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
15/09/17 11:20:13 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
15/09/17 11:20:14 INFO ObjectStore: Initialized ObjectStore
15/09/17 11:20:15 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 0.13.1aa
15/09/17 11:20:16 INFO HiveMetaStore: Added admin role in metastore
15/09/17 11:20:16 INFO HiveMetaStore: Added public role in metastore
15/09/17 11:20:17 INFO HiveMetaStore: No user is added in admin role, since config is empty
15/09/17 11:20:18 INFO SessionState: No Tez session required at this point. hive.execution.engine=mr.
15/09/17 11:20:18 INFO SparkILoop: Created sql context (with Hive support)..
SQL context available as sqlContext.

我在MySQL中建立了一个数据库。现在我想将它与 Spark 连接。 我使用 bin/sparkling-shell --jars mysql:mysql-connector-java:5.1.36 并得到警告:跳过远程 jar mysql:mysql-connector-java:5.1.36。

我下载了mysql-connector-java-5.1.36.tar.gz放在home/tong/sparkling-water还是不行。

如何包含 jdbc jar 文件? 我使用的是 Spark 1.4.1 有没有其他方法可以连接 Mysql 和 Spark?

最佳答案

bin/spark-shell --driver-class-path/home/tg/sparkling-water/mysql-connector-java-5.1.36-bin.jar

val jdbcDF = sqlContext.load("jdbc", Map("url"-> "jdbc:mysql://localhost:3306/employee?user=tg&password=********", "dbtable"->“员工”))

关于mysql - 如何包含 MySQL 连接器 jar,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/32633963/

相关文章:

scala - 如何在Spark中顺序处理两个RDD?

python - 在spark Dataframe中动态创建多列

amazon-web-services - 如何在Amazon Spark集群上启动4个实例?

hadoop - Spark setAppName 未出现在 Hadoop 运行应用程序 UI 中

mysql - 如何在 MySQL 的 PROCEDURE 中使用新插入记录的 id?

mysql - 在 shell 中使用变量将密码传递给 mysql_config_editor

php - MySQL 和 PHP 代码允许空表单字段数据

java - Spark CSV Reader 引用数字

mysql - 记住我表索引

java - hibernate 不返回新插入的记录