scala - 连接 hive 和spark时发生异常HDFS上的根暂存目录:/tmp/hive应该是可写的。当前权限是:rwxrwxr-x

Question

我正在使用Spark 2.4.5
hive 3.1.2，
Hadoop 3.2.1。
在 hive 中运行 Spark 时，出现以下异常，

Exception in thread "main" org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwxrwxr-x; 

这是我的源代码，

package com.spark.hiveconnect

import java.io.File

import org.apache.spark.sql.{Row, SaveMode, SparkSession}

object sourceToHIve {
  case class Record(key: Int, value: String)
  def main(args: Array[String]){
    val warehouseLocation = new File("spark-warehouse").getAbsolutePath

    val spark = SparkSession
      .builder()
      .appName("Spark Hive Example").master("local")
      .config("spark.sql.warehouse.dir", warehouseLocation)
      .enableHiveSupport()
      .getOrCreate()

    import spark.implicits._
    import spark.sql

    sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING) USING hive")
    sql("LOAD DATA LOCAL INPATH '/usr/local/spark3/examples/src/main/resources/kv1.txt' INTO TABLE src")
    sql("SELECT * FROM src").show()

    spark.stop()
  }

}

这是我的sbt文件

name := "SparkHive"

version := "0.1"

scalaVersion := "2.12.10"

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.4.5"
// https://mvnrepository.com/artifact/org.apache.spark/spark-sql
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.5"
// https://mvnrepository.com/artifact/mysql/mysql-connector-java
libraryDependencies += "mysql" % "mysql-connector-java" % "8.0.19"
libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.4.5"

如何解决这个问题？
在观察控制台的同时，我也看到了此声明，这就是为什么我遇到此问题的原因。

20/05/28 14:03:04 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(UDHAV.MAHATA); groups with view permissions: Set(); users  with modify permissions: Set(UDHAV.MAHATA); groups with modify permissions: Set()

谁能帮我？

谢谢!

Answer 1

您可以尝试从 shell 更改权限

hdfs dfs -chmod -R 777 /tmp/hive

或在正确的权限下与其他用户或应用程序一起运行您的Spark流程。

您可以使用以下方法更改目录或文件的所有者:

-chown [-R] [OWNER][:[GROUP]] PATH... :
  Changes owner and group of a file. This is similar to the shell's chown command
  with a few exceptions.

  -R  modifies the files recursively.