string - 如何在mapreduce中解决mapper设置方法给定的字符串值的不规则行为？

Question

我是MapReduce的新手，正在学习设置方法的实现。配置给定的新字符串值可以正确打印，但是当我尝试进一步处理它时，字符串的初始值起作用。我知道字符串是不可变的，但是它应该提供当前指向其他方法的值。

public class EMapper extends Mapper<LongWritable, Text, Text, Text> {

    String wordstring = "abcd"; //initialized wordstring with "abcd"


    public void setup(Context context) {
        Configuration config = new Configuration(context.getConfiguration());
        wordstring = config.get("mapper.word"); // As string is immutable,
        // wordstring should now point to
        // value given by mapper.word
        //Here mapper.word="ankit" by 
        //using -D in hadoop command

    }

    String def = wordstring;
    String jkl = String.valueOf(wordstring); //tried to copy current value 
    //but 
    //string jkl prints the initial 
    /value.

    public void map(LongWritable key, Text value, Context context)
    throws InterruptedException, IOException {
        context.write(new Text("wordstring=" + wordstring + "   " + "def=" + 
                def),
            new Text("jkl=" + jkl));
    }
}


public class EDriver extends Configured implements Tool {

    private static Logger logger = LoggerFactory.getLogger(EDriver.class);


    public static void main(String[] args) throws Exception {
        logger.info("Driver started");

        int res = ToolRunner.run(new Configuration(), new EDriver(), args);
        System.exit(res);
    }

    public int run(String[] args) throws Exception {
        if (args.length != 2) {
            System.err.printf("Usage: %s  needsarguments",
                getClass().getSimpleName());
            return -1;
        }
        Configuration conf = getConf();
        Job job = new Job(conf);
        job.setJarByClass(EDriver.class);
        job.setJobName("E Record Reader");

        job.setMapperClass(EMapper.class);
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(Text.class);
        job.setReducerClass(EReducer.class);
        job.setNumReduceTasks(0);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(NullWritable.class);

        FileInputFormat.addInputPath(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));

        job.setInputFormatClass(ExcelInputFormat.class);

        return job.waitForCompletion(true) ? 0 : 1;
    }

} 

我期望输出是

   wordstring=ankit   def=ankit   jkl=ankit

实际输出为

   wordstring=ankit   def=abcd    jkl=abcd

Answer 1

这与字符串的可变性无关，与代码执行顺序无关。

仅在执行任何类级别的命令之后，才会调用setup方法。您编写代码的顺序没有任何改变。如果要按照实际执行的顺序重写代码的顶部，您将需要:

public class EMapper extends Mapper<LongWritable, Text, Text, Text> {
    String wordstring = "abcd";
    String jkl = String.valueOf(wordstring);

    public void setup(Context context) {
        Configuration config = new Configuration(context.getConfiguration());
        wordstring = config.get("mapper.word"); //By the time this is called, jkl has already been assigned to "abcd"
    }

因此jkl仍然是abcd也就不足为奇了。您应该在jkl方法中设置setup，如下所示:

public class EMapper extends Mapper<LongWritable, Text, Text, Text> {
    String wordstring;
    String jkl;

    public void setup(Context context) {
        Configuration config = new Configuration(context.getConfiguration());
        wordstring = config.get("mapper.word");
        jkl = wordstring;
        //Here, jkl and wordstring are both different variables pointing to "ankit"
    }

    //Here, jkl and wordstring are null, as setup(Context context) has not yet run

    public void map(LongWritable key, Text value, Context context)
        throws InterruptedException, IOException {
        //Here, jkl and wordstring are both different variables pointing to "ankit"
        context.write(new Text("wordstring=" + wordstring),
            new Text("jkl=" + jkl));
    }

当然，您实际上并不需要jkl，您可以直接使用wordstring。