hadoop - 我可以使用Text作为值写入MapReduce中的上下文吗

Question

我有一个方案来计算map reduce中两列的平均值。所以我所做的是，我已经使用映射器从文件中获取了值并将它们连接为Text，然后尝试将它们写入如下所示的Context中。

class TestMapper extends Mapper<LongWritable, Text, Text, Text> {
  private Text outputKey;
  private Text outputVal;


  @Override
  public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
  
  //more code here

context.write(outputkey,OutputVal);


    
  }
}

Answer 1

您应该在此处使用自定义数据类型，例如一个TextPair类，该类具有两个Text元素来存储所需的数据。下面是一个示例代码，用于以mapper上下文的值输出一对String。

// Mapper's map code
protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Text, TextPair>.Context context)
        throws IOException, InterruptedException {

    String line = value.toString();
    String year = line.substring(15, 19);
    int airTemperature;
    if (line.charAt(87) == '+') { // parseInt doesn't like leading plus signs
      airTemperature = Integer.parseInt(line.substring(88, 92));
    } else {
      airTemperature = Integer.parseInt(line.substring(87, 92));
    }
    String quality = line.substring(92, 93);
    if (airTemperature != MISSING && quality.matches("[01459]")) {
        System.out.println("Year "+year+" "+airTemperature);
      context.write(new Text(year), new TextPair(String.valueOf(airTemperature),1));
    }

//文本对-下面的自定义数据类型代码

public class TextPair implements WritableComparable<TextPair> {

private Text first;
private Text second;

//Default constructor is a must
public TextPair() {
    this.first=new Text();
    this.second=new Text();
}

public TextPair(String first,int second) {
    try {
        this.first=new Text(first);
        this.second=new Text(String.valueOf(second));
    }catch(Exception ex) {
        System.out.println("Exception occurred "+ex.getCause());
    }

}

// Other methods such as compare, equals, hashcode, write, readFields etc implementation also needs to done

public Text getFirst() {
    return first;
}

public Text getSecond() {
    return second;
}

@Override
public String toString() {
    return this.first+"\t"+this.second+"\t";
}

}

如果您还需要更多详细信息，请引用《 Hadoop权威指南》。希望这可以帮助。