mysql - 如何将在 Intellij 中运行的 Spark 作业连接到本地 mysql hive metastore？

Question

我正在尝试使用 Intellij 测试需要创建配置单元表的 spark scala 代码。我已经在我的 MAC 上本地使用 mysql 驱动程序安装了配置单元。我可以使用

从 spark-shell 创建一个配置单元表

sqlContext.sql("CREATE TABLE IF NOT EXISTS employee(id INT, name STRING, age INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n'")

但在 Intellij 中的 scala 程序中，相同的命令即使成功运行到完成，也无法实际创建显示在 hive metastore 上的任何表。

val spark = SparkSession.builder
        .appName("BiddingExternalTable")
        .master("local")
        .enableHiveSupport()
        .getOrCreate()

      spark.sqlContext.sql("CREATE TABLE IF NOT EXISTS employeeExternal(id INT, name STRING, age INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n'")

查看控制台输出，Intellij 中的 spark session 仍在使用默认的 DERBY 元存储。

19/05/02 17:40:06 INFO SharedState: Warehouse path is 'file:/Users/sichu/src/MktDataSSS/spark-warehouse/'.
19/05/02 17:40:07 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
19/05/02 17:40:09 INFO HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
19/05/02 17:40:09 INFO HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
19/05/02 17:40:09 INFO ObjectStore: ObjectStore, initialize called
19/05/02 17:40:10 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
19/05/02 17:40:10 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
19/05/02 17:40:10 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
19/05/02 17:40:11 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
19/05/02 17:40:11 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
19/05/02 17:40:11 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
19/05/02 17:40:11 INFO Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
19/05/02 17:40:11 INFO Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
19/05/02 17:40:11 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY

尽管我已将 JDBC 驱动程序(及其文件夹)添加到 CLASSPATH。我还将 hive-site.xml 文件放在 hadoop conf 目录中。这个 hive-site.xml 已被 spark-shell 成功获取，但在从 Intellij 内部运行 scala 程序时却没有。

有人可以帮助我将 Intellij 中的 spark 作业连接到我在本地计算机上设置的 mysql hive metastore。谢谢!

Answer 1

您应该为您分配 Metastore 位置

val spark = SparkSession
      .builder()
      .master("yarn")
      .appName("Test Hive Support")
      .config("hive.metastore.uris", "jdbc:mysql://localhost/metastore")
       //or .config("hive.metastore.uris", "thrift://localhost:9083")
      .enableHiveSupport
      .getOrCreate();