java - 在mapreduce程序中未调用reducer

Question

我在Mapreduce程序上写了一个简单的扩展，发现我的代码仅显示Map()的输出。映射作业在eclipse中运行，没有任何错误，但不调用reduce()。

这是我的map():

public static class KVMapper 
  extends Mapper<Text, Text, IntWritable, Text>{
//        extends Mapper<Text, Text, Text, IntWritable>{
    private final static IntWritable one = new IntWritable(1);
    private String word;// = new Text();
    private IntWritable iw;
    private final LongWritable val = new LongWritable();
    public void map(Text key, Text value , Context context
                    ) throws IOException, InterruptedException {

      iw = new IntWritable(Integer.parseInt(value.toString()));
      System.out.println(value +" hello , world  " +key );
      context.write(iw, key);
      }
    }

降低()

public static class KVReducer 
       extends Reducer<IntWritable,Text,IntWritable, Text> {

      KVReducer(){

          System.out.println("Inside reducer");
      }
        public void reduce(IntWritable key, Text value, 
                       Context context
                       ) throws IOException, InterruptedException {
      System.out.println(value +" hello2 , world  " +key );
      context.write(key, value);
    }
  }

主要()

public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    conf.set("mapreduce.input.keyvaluelinerecordreader.key.value.separator", "\t");
    //conf.set("mapreduce.input.keyvaluelinerecordreader.key.value.separator",",");
    String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
    if (otherArgs.length < 2) {
      System.err.println("Usage: wordcount <in> [<in>...] <out>");
      System.exit(2);
    }
    Job job = new Job(conf, "word desc");
    job.setInputFormatClass(KeyValueTextInputFormat.class);
    job.setJarByClass(WordDesc.class);
    job.setMapperClass(KVMapper.class);
    job.setCombinerClass(KVReducer.class);
    job.setReducerClass(KVReducer.class);
    job.setMapOutputKeyClass(IntWritable.class);
    job.setMapOutputValueClass(Text.class);
    job.setOutputKeyClass(IntWritable.class);
    job.setOutputValueClass(Text.class);
    for (int i = 0; i < otherArgs.length - 1; ++i) {
      FileInputFormat.addInputPath(job, new Path(otherArgs[i]));
    }
    FileOutputFormat.setOutputPath(job,
      new Path(otherArgs[otherArgs.length - 1]));
    System.exit(job.waitForCompletion(true) ? 0 : 1);
  }

输入样本:

1500s   1
1960s   1
Aldus   1

在我期望映射器反转键和值对的同时，该程序的示例输出

1500s   1
1960s   1
Aldus   1

不知道为什么在上面的代码中没有调用reduce()

Answer 1

您没有覆盖reduce()类的Reducer方法。

对于您的情况，其签名应类似于public void reduce(IntWritable key, Iterable<Text> values,Context context)
这是更新的KVReducer

public static class KVReducer 
       extends Reducer<IntWritable,Text,IntWritable, Text> {

      KVReducer(){

          System.out.println("Inside reducer");
      }

      public void reduce(IntWritable key, Iterable<Text> values,Context context) throws IOException, InterruptedException {
        for(Text value: values){}
          System.out.println(value +" hello2 , world  " +key );
          context.write(key, value);
        }
      }
}