hadoop - 如何设置OutputCommitter配置？

Question

我有一个mapreduce代码，它使用JobClient提交作业。我不断收到此空指针错误堆栈

12/12/10 12:42:44 INFO mapred.LocalJobRunner: OutputCommitter set in config null    
Exception in thread "main" java.lang.NullPointerException
        at org.apache.hadoop.mapred.JobClient$NetworkedJob.<init>(JobClient.java:226)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:924)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:844)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:844)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:818)
        at FTPIF.run(FTPIF.java:193)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at FTPIF.main(FTPIF.java:273)

引起我这个错误的代码是

JobClient j = new JobClient();
        j.init(conf);
        RunningJob check = j.submitJob(conf);

有任何想法吗 ？

Answer 1

我也有这个问题。
但是使用j.runJob(conf);效果很好。
您也可以使用静态函数JobClient.runJob(conf);。
我不知道为什么，然后阅读了JobClient的源代码，并找到了静态函数runJob的实现:

public static RunningJob runJob(JobConf job) throws IOException {
    JobClient jc = new JobClient(job);
    RunningJob rj = jc.submitJob(job);
    try {
        if (!jc.monitorAndPrintJob(job, rj)) {
            throw new IOException("Job failed!");
        }
    } catch (InterruptedException ie) {
        Thread.currentThread().interrupt();
    }
    return rj;
}

因此，我以这种方式使用submitJob函数:

JobClient jcli = new JobClient(jconf);
RunningJob rJob = jcli.submitJob(jconf);
while (true){
    Thread.sleep(5000);
    System.out.println(rJob);
    if (rJob.isComplete())
        break;
}

我工作!!也许这对您也有用。