java - 尝试将 LZO 压缩与 MapReduce 结合使用

Question

我想在 MapReduce 中使用 LZO 压缩，但在运行 MapReduce 作业时出现错误。我正在使用带有 Java 程序的 Ubuntu。我只是想在我的本地机器上运行它。我最初的错误是

ERROR lzo.GPLNativeCodeLoader: Could not load native gpl library

线下

ERROR lzo.LzoCodec: Cannot load native-lzo without native-hadoop

然后

java.lang.RuntimeException: native-lzo library not available

我遵循了一些关于如何下载和配置文件以使用 LZO 压缩的在线和文本说明。这里可以在lib文件夹下看到我的hadoop-lzo jar文件

我已经更改了我的配置。这是我的 core-site.xml

<configuration>
<property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
</property>
<property>
    <name>io.compression.codecs</name>
        <value>com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec</value>
</property>

<property>
    <name>io.compression.codec.lzo.class</name>
    <value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
<property>
    <name>mapred.output.compress</name>
    <value>true</value>
</property>

<property>
    <name>mapred.output.compression.codec</name>
    <value>com.hadoop.compression.lzo.LzopCodec</value>
</property>
</configuration>

和我的 mapred-site.xml

<configuration>
<property>
    <name>mapred.job.tracker</name>
    <value>localhost:9001</value>
</property>
<property>
    <name>mapred.compress.map.output</name>
    <value>true</value>
</property>
<property>
    <name>mapred.map.output.compression.codec</name>
    <value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
<property>
    <name>mapred.child.env</name>
    <value>JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:/home/matthew/hadoop/lib/native/lib/lib</value>
</property>
</configuration>

我还使用这些行在同一个 conf 文件夹中修改了我的 hadoop-env.sh

export HADOOP_CLASSPATH=/home/hadoop/lib/hadoop-lzo-0.4.13.jar
export JAVA_LIBRARY=/home/hadoop/lib/native/lib/lib

如果你对/home/hadoop/lib/native/lib/lib中的内容感兴趣的话

值得一提的是，这是我的 Driver 类，它执行所有压缩

//import java.net.URI;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
//import org.apache.hadoop.filecache.DistributedCache;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
//import org.apache.hadoop.io.compress.CompressionCodec;
import com.hadoop.compression.lzo.LzopCodec;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
//import org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;

public class FuzzyJoinDriver extends Configured implements Tool{

public static void main(String[] args) throws Exception {     
    int exitCode = ToolRunner.run(new Configuration(), new  FuzzyJoinDriver(),args);
System.exit(exitCode);
}
@Override
public int run(String[] args) throws Exception {
    if (args.length != 2) {
          System.err.println("Usage: FuzzyJoinDriver <input path> <output path>");
          System.exit(-1);
        }

        Configuration conf = new Configuration();
        //Used to compress map output
        //conf.setBoolean("mapred.compres.map.output", true);
        //conf.setClass("mapred.map.output.compression.code", GzipCodec.class, CompressionCodec.class);
        Job job = new Job(conf);
        job.setJarByClass(FuzzyJoinDriver.class);
        job.setJobName("Fuzzy Join");

        //Distributed Cache
        //DistributedCache.addCacheFile(new URI("/cache/ReducerCount.txt"),  job.getConfiguration());


        FileInputFormat.addInputPath(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));

        job.setMapperClass(FuzzyJoinMapper.class);


        job.setReducerClass(FuzzyJoinReducer.class);
        job.setMapOutputKeyClass(IntWritable.class);
        job.setMapOutputValueClass(RelationTuple.class);


        job.setPartitionerClass(JoinKeyPartitioner.class);


        job.setOutputKeyClass(NullWritable.class);
        job.setOutputValueClass(Text.class);

        //Used to compress output from reducer
        FileOutputFormat.setCompressOutput(job,  true);
        FileOutputFormat.setOutputCompressorClass(job, LzopCodec.class);
        //SequenceFileOutputFormat.setOutputCompressionType(job, org.apache.hadoop.io.SequenceFile.CompressionType.BLOCK);

        return(job.waitForCompletion(true) ? 0 : 1);
    }
}

它编译没有错误。

我担心我对配置和其他步骤的不了解导致我走错了路，也许我错过了一些对那些比我更了解这件事的人来说简单的东西。感谢您做到这一点。我知道这是一篇冗长的文章。

Answer 1

您需要在 mapred-site.xml 中设置以下属性:

<property>
    <name>mapred.compress.map.output</name>
    <value>true</value>
</property>

<property>
    <name>mapred.map.output.compression.codec</name>
    <value>org.apache.hadoop.io.compress.LzoCodec</value>
</property>

如果您想在每个作业的基础上执行此操作，您还可以通过以下方式之一直接在您的代码中实现它:

conf.set("mapred.compress.map.output", "true")
conf.set("mapred.map.output.compression.codec", "org.apache.hadoop.io.compress.LzoCodec");

或

jobConf.setMapOutputCompressorClass(LzoCodec.class);