问题陈述:我需要传递一个字符串列表并使用spark java中的过滤器函数应用like运算符。
原因: like 运算符考虑单个字符串实体,因此代码中存在很多性能问题,因为我们需要应用于更大的数据集来克服此问题 我需要传递字符串列表并使用过滤功能应用类似运算符。
请让我知道如何使用 like 来应用列表。因为我需要找到相似的 ID 模式
JavaSparkContext sc = new JavaSparkContext(new SparkConf().setAppName("SparkJdbcDs").setMaster("local[*]"));
SQLContext sqlContext = new SQLContext(sc);
SparkSession spark = SparkSession.builder().appName("JavaTokenizerExample").getOrCreate();
List<Row> data = Arrays.asList(
RowFactory.create("J40504", "CRC Industries"),
RowFactory.create("K630-0746777","Dixon value"),
RowFactory.create("K444-4444","3M INdustries"),
RowFactory.create("4333444","3M INdustries"),
RowFactory.create("566-655","3M INdustries"),
RowFactory.create("4444888","3M INdustries"),
RowFactory.create("P477-7444","3M INdustries"),
RowFactory.create("566655","Dixon coupling valve"));
// In real time we have large dataset
StructType schema = new StructType(new StructField[] {new StructField("label1", DataTypes.StringType, false,Metadata.empty()),
new StructField("sentence1", DataTypes.StringType, false,Metadata.empty()) });
Dataset<Row> sentenceDataFrame = spark.createDataFrame(data, schema);
List<String> listStrings = new ArrayList<String>();
listStrings.add("40504");
listStrings.add("630-0746");
listStrings.add("477-7444");
listStrings.add("444-4444");
// In real time we have large list of string to be compared with
sentenceDataFrame.show();
System.out.println("Array list :"+listStrings);
for(int i=0;i<listStrings.size();i++){
sentenceDataFrame=sentenceDataFrame.filter(col("label1").like("%"+listStrings.get(i)+"%"));
}
sentenceDataFrame.show();
最佳答案
第一个解决方案
您可以构建表达式并仅过滤数据集一次,而不是迭代数据集 N(其中 N 是 listStrings 的大小)次数:
StringBuilder expressionBuilder = new StringBuilder();
String separator = "";
for (String s : listStrings) {
expressionBuilder.append(separator + " label1 LIKE '%" + s + "%'");
separator = " OR ";
}
String expression = expressionBuilder.toString();
sentenceDataFrame = sentenceDataFrame.filter(expression);
第二种解决方案
我们可以将 listStrings 加载到数据集中:
StructType schemaList = new StructType(new StructField[]{new StructField("labelToFind", DataTypes.StringType, false, Metadata.empty())});
List<Row> listStrings = Arrays.asList(
RowFactory.create("40504"),
RowFactory.create("630-0746"),
RowFactory.create("477-7444"),
RowFactory.create("444-4444"));
Dataset<Row>listDataset = sqlContext.createDataFrame(listStrings, schemaList);
然后我们可以连接两个数据集以过滤行:
sentenceDataFrame = sentenceDataFrame.join(listDataset ,sentenceDataFrame.col("label1").contains(listDataset.col("labelToFind"))).select("label1","sentence1");
关于java - 在类似运算符上应用字符串列表,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/44135544/