java - hadoop 中多种输入格式作为单一输入格式

Question

我面临以下情况。请帮助我。我使用hadoop Mapreduce 来处理XML 文件。

通过引用此网站，我可以滑动我的记录 https://gist.github.com/sritchie/808035 但是当 XML 文件的大小大于 block 大小时，我无法获得正确的值 所以我需要读取整个文件 为此我得到了这个链接

https://github.com/pyongjoo/MapReduce-Example/blob/master/mysrc/XmlInputFormat.java

但现在的问题是如何将两个inputformat实现为单个inputformat

请尽快帮助我 谢谢

更新

public class XmlParser11
{

        public static class XmlInputFormat1 extends TextInputFormat {

        public static final String START_TAG_KEY = "xmlinput.start";
        public static final String END_TAG_KEY = "xmlinput.end";

        @Override
    protected boolean isSplitable(JobContext context, Path file) {
        return false;
        }


        public RecordReader<LongWritable, Text> createRecordReader(InputSplit split, TaskAttemptContext context) {
            return new XmlRecordReader();
        }

        /**
         * XMLRecordReader class to read through a given xml document to output
         * xml blocks as records as specified by the start tag and end tag
         *
         */


        public static class XmlRecordReader extends RecordReader<LongWritable, Text> {
            private byte[] startTag;
            private byte[] endTag;
            private long start;
            private long end;
            private FSDataInputStream fsin;
            private DataOutputBuffer buffer = new DataOutputBuffer();

            private LongWritable key = new LongWritable();
            private Text value = new Text();
            @Override
            public void initialize(InputSplit split, TaskAttemptContext context)
                    throws IOException, InterruptedException {
                Configuration conf = context.getConfiguration();
                startTag = conf.get(START_TAG_KEY).getBytes("utf-8");
                endTag = conf.get(END_TAG_KEY).getBytes("utf-8");
                FileSplit fileSplit = (FileSplit) split;

但不工作

Answer 1

使用isSplitable属性指定不分割文件(即使达到 block 大小)。当您想要确保大文件应由单个映射器处理时，通常使用此方法。

public class XmlInputFormat extends FileInputFormat {
@Override
 protected boolean isSplitable(JobContext context, Path file) {
 return false;
}

@Override
 public RecordReader<LongWritable, Text> createRecordReader(InputSplit split,TaskAttemptContext context)
 throws IOException {
  // return your version of XML record reader
 }
}

或者，您也可以使用以下方法增加每个拆分的 block 大小:

// Set the maximum split size
setMaxSplitSize(MAX_INPUT_SPLIT_SIZE);