apache-spark - 为什么在 Spark-Shell 中导入 SparkSession 失败并显示 "object SparkSession is not a member of package org.apache.spark.sql"？

Question

我在我的虚拟机 Cloudera 机器上使用 Spark 1.6.0。

我正在尝试从 Spark shell 将一些数据输入到 Hive 表中。 为此，我尝试使用 SparkSession。但下面的导入不起作用。

scala> import org.apache.spark.sql.SparkSession
<console>:33: error: object SparkSession is not a member of package org.apache.spark.sql
         import org.apache.spark.sql.SparkSession

如果没有这个，我就无法执行这个语句:

val spark = SparkSession.builder.master("local[2]").enableHiveSupport().config("hive.exec.dynamic.partition","true").config("hive.exec.dynamic.partition.mode", "nonstrict").config("spark.sql.warehouse.dir", warehouseLocation).config("hive.metastore.warehouse.dir","/user/hive/warehouse").getOrCreate()
<console>:33: error: not found: value SparkSession
         val spark = SparkSession.builder.master("local[2]").enableHiveSupport().config("hive.exec.dynamic.partition","true").config("hive.exec.dynamic.partition.mode", "nonstrict").config("spark.sql.warehouse.dir", warehouseLocation).config("hive.metastore.warehouse.dir","/user/hive/warehouse").getOrCreate()

谁能告诉我我在这里犯了什么错误？

Answer 1

SparkSession is available as of Spark 2.0因此您应该改用 SQLContext (或将您的 Spark 升级到最新最好的 2.1.1 )。

引用 Spark 1.6.0 的 Starting Point: SQLContext :

The entry point into all functionality in Spark SQL is the SQLContext class, or one of its descendants.

In addition to the basic SQLContext, you can also create a HiveContext, which provides a superset of the functionality provided by the basic SQLContext.