就像标题所说,我想用phoenix保存我的DataFrame。
我在intellij IDEA上运行的scala中有一个 Spark 代码。
这很简单:
import org.apache.spark.sql.SparkSession
import org.apache.phoenix.spark._
object MainTest extends App {
val sparkSession = SparkSession.builder()
.config("spark.sql.warehouse.dir", "file:///c:/tmp/spark-warehouse")
.master("local[*]")
.appName("spark-to-hbase")
.getOrCreate()
val sc = sparkSession.sparkContext
val sqlC = sparkSession.sqlContext
import sqlC.implicits._
val myRdd = sc.parallelize(List(("a",1), ("b", 2)))
myRdd.collect.foreach(println)
val myDf = myRdd.toDF("column1", "column2")
myDf.show()
myDf.saveToPhoenix("MY_TABLE", zkUrl = Some("localhost:16000"))
}
我也有一个HBase数据库在端口16000上的同一台PC上运行。
问题是线
myDf.saveToPhoenix("MY_TABLE", zkUrl = Some("localhost:16000"))
抛出此异常:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/types/DataType at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.apache.phoenix.spark.DataFrameFunctions$$anonfun$getFieldArray$2.apply(DataFrameFunctions.scala:72) at org.apache.phoenix.spark.DataFrameFunctions$$anonfun$getFieldArray$2.apply(DataFrameFunctions.scala:72) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186) at org.apache.phoenix.spark.DataFrameFunctions.getFieldArray(DataFrameFunctions.scala:72) at org.apache.phoenix.spark.DataFrameFunctions.saveToPhoenix(DataFrameFunctions.scala:35) at MainTest$.delayedEndpoint$MainTest$1(MainTest.scala:23) at MainTest$delayedInit$body.apply(MainTest.scala:8) at scala.Function0$class.apply$mcV$sp(Function0.scala:34) at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12) at scala.App$$anonfun$main$1.apply(App.scala:76) at scala.App$$anonfun$main$1.apply(App.scala:76) at scala.collection.immutable.List.foreach(List.scala:381) at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35) at scala.App$class.main(App.scala:76) at MainTest$.main(MainTest.scala:8) at MainTest.main(MainTest.scala) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.types.DataType at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 33 more
据我了解,java似乎找不到某些类。
我能做什么 ?
我的SBT:
name := "spark-to-hbase"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"org.apache.hadoop" % "hadoop-mapreduce-client-core" % "2.3.0",
"org.apache.phoenix" % "phoenix-core" % "4.11.0-HBase-1.3",
"org.apache.spark" % "spark-core_2.11" % "2.1.1",
"org.apache.spark" % "spark-sql_2.11" % "2.1.1",
"org.apache.phoenix" % "phoenix-spark" % "4.11.0-HBase-1.3"
)
最佳答案
方法saveToPhoenix
采用RDD。对于数据框,请使用save
(有关更多详细信息,请参见此doc):
myDf.save("org.apache.phoenix.spark", SaveMode.Overwrite,
Map("MY_TABLE" -> "OUTPUT_TABLE", "zkUrl" -> "localhost:16000"))
关于java - 使用phoenix在Hbase上保存数据框,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/44850784/