我有 1 个 MR 作业,它的输出如下:
128.187.140.171,11
129.109.6.54,27
129.188.154.200,44
129.193.116.41,5
129.217.186.112,17
在第 2 个 MR 作业的映射器代码中,我正在这样做;
public void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
// Parse the input string into a nice map
// System.out.println(value.toString());
if (value.toString().contains(",")) {
System.out.println("Inside");
String[] arr = value.toString().split(",");
if (arr.length > 1) {
System.out.println(arr[1]);
context.write(new Text(arr[1]), new Text(arr[0]));
}
}
打印语句的输出是正确的:
Inside
11
Inside
27
但是 context.write 一直显示以下输出:
1,slip4068.sirius.com
1,hstar.gsfc.nasa.gov
1,ad11-010.compuserve.com
1,slip85-2.co.us.ibm.net
1,stimpy.actrix.gen.nz
1,j14.ktk1.jaring.my
1,ad08-009.compuserve.com
为什么我的 Keys 一直是 1? 这是我的驱动程序代码:
public int run(String[] args) throws Exception {
// TODO Auto-generated method stub
Configuration conf = getConf();
conf.set("mapreduce.output.textoutputformat.separator", ",");
Job job = new Job(conf, "WL Demo");
job.setJarByClass(WLDemo.class);
job.setMapperClass(WLMapper1.class);
job.setReducerClass(WLReducer1.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
Path in = new Path(args[0]);
Path out = new Path(args[1]);
Path out2 = new Path(args[2]);
FileInputFormat.setInputPaths(job, in);
FileOutputFormat.setOutputPath(job, out);
boolean succ = job.waitForCompletion(true);
if (!succ) {
System.out.println("Job1 failed, exiting");
return -1;
}
Job job2 = new Job(conf, "top-k-pass-2");
FileInputFormat.setInputPaths(job2, out);
FileOutputFormat.setOutputPath(job2, out2);
job2.setJarByClass(WLDemo.class);
job2.setMapperClass(WLMapper2.class);
// job2.setReducerClass(Reducer1.class);
job2.setInputFormatClass(TextInputFormat.class);
job2.setMapOutputKeyClass(Text.class);
job2.setMapOutputValueClass(Text.class);
job2.setNumReduceTasks(1);
succ = job2.waitForCompletion(true);
if (!succ) {
System.out.println("Job2 failed, exiting");
return -1;
}
return 0;
}
如何在我的第二个 MR 作业的输出键中获得正确的值?
最佳答案
将 job2.setNumReduceTasks(1)
更改为 job2.setNumReduceTasks(0)
。因为它正在运行一个将输出键作为 1 的身份缩减器,您应该将 1 作为 map1 输出中某些记录的键。
关于hadoop - 为什么 Context.Write 没有按预期工作 - Hadoop Map reduce,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/32663967/