Java - 必须在您的配置中设置主 URL

Question

我正在尝试在 apache Spark 中运行一些算法。我正进入(状态 Java - 即使我设置了配置，也必须在您的配置错误中设置主 URL。

SparkSession spark = SparkSession.builder().appName("Sp_LogistcRegression").config("spark.master", "local").getOrCreate();

这是我使用的代码

import java.io.FileOutputStream;
import java.io.IOException;
import java.io.ObjectOutputStream;

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.ml.classification.LogisticRegression;
import org.apache.spark.ml.classification.LogisticRegressionModel;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.mllib.util.MLUtils;

public class Sp_LogistcRegression {
    public void trainLogisticregression(String path, String model_path) throws IOException {
        //SparkConf conf = new SparkConf().setAppName("Linear Regression Example");


    //  JavaSparkContext sc = new JavaSparkContext(conf);
        SparkSession spark = SparkSession.builder().appName("Sp_LogistcRegression").config("spark.master", "local").getOrCreate();
        Dataset<Row> training =  spark.read().option("header","true").csv(path);
         System.out.print(training.count());

        LogisticRegression lr = new LogisticRegression().setMaxIter(10).setRegParam(0.3);

        // Fit the model
        LogisticRegressionModel lrModel = lr.fit(training);
        lrModel.save(model_path);



        spark.close();

    }

}

这是我的测试用例:

import java.io.File;

import org.junit.Test;

public class Sp_LogistcRegressionTest {
    Sp_LogistcRegression spl =new Sp_LogistcRegression ();




    @Test
        public void test() {

            String filename = "datas/seg-large.csv";
            ClassLoader classLoader = getClass().getClassLoader();
            File file1 = new File(classLoader.getResource(filename).getFile());
            spl. trainLogisticregression( file1.getAbsolutePath(), "/tmp");

        }

    }

为什么我会收到此错误？我在这里检查了解决方案 Spark - Error "A master URL must be set in your configuration" when submitting an app 它不起作用。 有什么线索吗？

Answer 1

你的

SparkSession spark = SparkSession.builder().appName("Sp_LogistcRegression").config("spark.master", "local").getOrCreate();

应该是

SparkSession spark = SparkSession.builder().appName("Sp_LogistcRegression").master("local").getOrCreate();

或者

当你运行时，你需要

spark-submit --class mainClass --master local yourJarFile