java - Hadoop mapreduce-java.io.IOException : Job failed

Question

我在尝试执行 hadoop mapreduce 程序时遇到以下异常。

java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:865) at com.vasa.books.BookDriver.main(BookDriver.java:37)

BookDriver.java

package com.vasa.books;

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.TextInputFormat;
import org.apache.hadoop.mapred.TextOutputFormat;

public class BookDriver {

    public static void main(String args[]) {
        // TODO Auto-generated method stub
        JobClient client=new JobClient();
        JobConf conf=new JobConf(com.vasa.books.BookDriver.class);

        conf.setJobName("booknamefind");


        conf.setOutputKeyClass(Text.class);
        conf.setOutputValueClass(IntWritable.class);

        conf.setMapperClass(com.vasa.books.bookmapper.class);
        conf.setReducerClass(com.vasa.books.bookreducer.class);

        conf.setInputFormat(TextInputFormat.class);
        conf.setOutputFormat(TextOutputFormat.class);

        FileInputFormat.setInputPaths(conf,new Path(args[0]));
        FileOutputFormat.setOutputPath(conf, new Path(args[1]));

        client.setConf(conf);
        try{
        JobClient.runJob(conf);
        }catch(Exception e){
        e.printStackTrace();
        }

    }
}

BookMapper.java

package com.vasa.books;

import java.io.IOException;  

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.io.WritableComparable;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;

public class BookMapper extends MapReduceBase implements Mapper<LongWritable,Text,Text,IntWritable> {
    private final static IntWritable one=new IntWritable(1);
    public void map(LongWritable _key, Text value,
            OutputCollector<Text, IntWritable> output, Reporter reporter)
            throws IOException {
        // TODO Auto-generated method stub
        String Tempstring=value.toString();
        String[] singlebookdata=Tempstring.split("\",\"");
        output.collect(new Text(singlebookdata[3]), one);


    }

}

为什么会出现这个异常？

Answer 1

根据 JobClient 来源，JobClient.runJob() 调用返回 boolean 值的 JobClient.monitorAndPrintJob()。如果该 boolean 值为假(意味着作业失败)，它会打印出无用的错误消息“something failed!”你看到的。

要解决这个问题，您有两个选择:

1 -(更快)检查日志。 RunningJob 失败信息应该打印到日志中。

2 - 如果您不知道日志在哪里，没有启用日志记录，或者不想挖掘日志，您可以重写一些代码。我将不使用 JobClient.runJob()，而是执行与 runJob() 在您的代码中所做的等效的操作，这样当它失败时您会收到一条有用的错误消息.

  public static RunningJob myCustomRunJob(JobConf job) throws Exception {
    JobClient jc = new JobClient(job);
    RunningJob rj = jc.submitJob(job);
    if (!jc.monitorAndPrintJob(job, rj)) {
      throw new IOException("Job failed with info: " + rj.getFailureInfo());
    }
    return rj;
  }

我猜潜在的问题是没有找到 arg[0] 或 arg[1](您的输入或输出文件)。