在hadoop 0.20.2版本中,可以通过以下方式向jobconf添加输入/输出压缩:
jobConf.setBoolean("mapred.output.compress", true);
jobConf.setClass("mapred.output.compression.codec", BZip2Codec.class, CompressionCodec.class);
不建议使用jobConf,而应改用job。如何在其中添加压缩/解压缩?特别是,如何更改wordcount示例以输入bzip2文件:
public class WordCount {
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
Job job = new Job(conf, "Example Hadoop 0.20.1 WordCount");
job.setJarByClass(WordCount.class);
job.setMapperClass(TokenCounterMapper.class);
job.setReducerClass(TokenCounterReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
最佳答案
提交作业时,使用以下Configuration类
Configuration conf = new Configuration();
conf.set("mapred.output.compression.codec",
"org.apache.hadoop.io.compress.BZip2Codec");
Job job = new Job(conf);
关于compression - Hadoop 0.20.205 Job(而非JobConf)Bzip2压缩,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/8502715/