java - 在hadoop中用相同的键减去两个数字

Question

我有 2 个表格文件

文件 1:

key1 value1

键2值2

...

文件 2:

key1 value3

key2 value4

...

我想产生一个减少输出的形式

key1 (value1-value3)/value1

键2(值2-值4)/值2

我让 map 写下键，值前面加上一个字符，告诉它是 来自 file1 或 file2，但不确定如何编写 reduce 阶段

我的 map 方法是

public void map(LongWritable key,Text val,Context context) throws IOException,     InterruptedException
    {
        Text outputKey = new Text();
        Text outputValue = new Text();
        outputKey.set(key.toString());
        if ("A")
        {               
            outputValue.set("A,"+val);
        }
        else
        {
            outputValue.set("B," + val);
        }
        context.write(outputKey,  outputValue);
    }
}

Answer 1

它应该足够简单，因为您已经标记了它，尽管开始时有点困惑。我假设发出的值类似于 A23(对于 file1)和 B139(对于 file2)。片段:

public void reduce(Text key, Iterable<Text> values, Context context)
        throws IOException, InterruptedException {

    int diff = 0;
    int denominator = 1;
    for (Text val : values) {
        if (val.toString().startsWith("A")) {
            denominator = Integer.parseInt(val.toString().substring(1));
            diff += denominator;
        } else if (val.toString().startsWith("B")) {
            diff -= Integer.parseInt(val.toString().substring(1));
        } else {
            // This block shouldn't be reached unless malformed values are emitted
            // Throw an exception or log it
        }
    }
    diff /= denominator;
    context.write(key, new IntWritable(diff));
}

希望这会有所帮助。但我认为当 key1 和 key2 相等时，您的方法会严重失败。

更新
map 应该像下面这样才能与上面的 reducer 一起工作:

public void map(LongWritable key, Text val, Context context)
            throws IOException, InterruptedException {
        String fileName = ((FileSplit) context.getInputSplit()).getPath().getName();
        String[] keyVal = val.toString().split("\\s+");
        Text outputKey = new Text(keyVal[0]);
        Text outputValue = new Text();
        outputKey.set(key.toString());
        if ("fileA".equals(fileName)) {
            outputValue.set("A" + keyVal[1]);
        } else {
            outputValue.set("B" + keyVal[1]);
        }
        context.write(outputKey, outputValue);
    }