我已经在 eclipse 中编写了 Java 程序来连接 spark 中的两个表,但是在 package
附近出现错误
这是下面的错误
The type scala.reflect.api.TypeTags$TypeTag cannot be resolved. It is indirectly referenced from required .class files
这是我写的程序
package joins;
import org.apache.spark.SparkConf;
import org.apache.spark.SparkContext;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.sql.DataFrame;
import org.apache.spark.sql.hive.HiveContext;
public class Spark {
public static void main(String[] args) {
// TODO Auto-generated method stub
SparkConf conf = new SparkConf();
SparkContext sc = new SparkContext(conf);
HiveContext sqlContext = new org.apache.spark.sql.hive.HiveContext(sc);
sqlContext.sql(" use myown ");
DataFrame table_01 = sqlContext.sql("SELECT * FROM customer");
table_01.saveAsTable("spark_table_01");
sqlContext.cacheTable("spark_table_01");
DataFrame table_02 = sqlContext.sql("SELECT * FROM account");
table_02.saveAsTable("spark_table_02");
sqlContext.cacheTable("spark_table_02");
DataFrame table_join = sqlContext.sql(" SELECT a.* FROM customer a join account b on a.ssn=b.ssn ");
table_join.insertInto("customeraccount");
sqlContext.uncacheTable("spark_table_01");
sqlContext.uncacheTable("spark_table_02");
}
}
最佳答案
您的应用程序中似乎缺少 scala reflect jar 文件。下载scala-reflect.jar,放入类路径中,重新编译。
关于java - 将两个存在于 hive 中的 spark 表连接起来,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/33351234/