python - 使用 mrjob 处理 LZO 序列文件

Question

我正在用 mrjob 编写一个任务使用 Google Ngrams 数据计算各种统计数据:https://aws.amazon.com/datasets/8172056142375670

我使用制表符分隔文本中的未压缩数据子集在本地开发和测试了我的脚本。当我尝试运行该作业时，我收到此错误:

Traceback (most recent call last):
  File "ngram_counts.py", line 74, in <module>
    MRNGramCounts.run()
  File "/usr/lib/python2.6/dist-packages/mrjob/job.py", line 500, in run
    mr_job.execute()
  File "/usr/lib/python2.6/dist-packages/mrjob/job.py", line 509, in execute
    self.run_mapper(self.options.step_num)
  File "/usr/lib/python2.6/dist-packages/mrjob/job.py", line 574, in run_mapper
    for out_key, out_value in mapper(key, value) or ():
  File "ngram_counts.py", line 51, in mapper
    (ngram, year, _mc, _pc, _vc) = line.split('\t')
ValueError: need more than 2 values to unpack
(while reading from s3://datasets.elasticmapreduce/ngrams/books/20090715/eng-1M/5gram/data)

这大概是因为公共(public)数据集的压缩方案(来自上面的 URL 链接):

We store the datasets in a single object in Amazon S3. The file is in sequence file format with block level LZO compression. The sequence file key is the row number of the dataset stored as a LongWritable and the value is the raw data stored as TextWritable.

关于如何设置可以处理这些文件的工作流程有任何指导吗？我已经详尽地搜索了提示，但没有找到任何有用的东西......

(我是 mrjob 和 Hadoop 的相对新手。)

Answer 1

我终于明白了这一点。看起来 EMR 会为您处理 LZO 压缩，但对于序列文件格式，您需要将以下 HADOOP_INPUT_FORMAT 字段添加到 MRJob 类中:

class MyMRJob(MRJob):

    HADOOP_INPUT_FORMAT = 'org.apache.hadoop.mapred.SequenceFileAsTextInputFormat'

    def mapper(self, _, line):
        # mapper code...

    def reducer(self, key, value):
        # reducer code...

还有另一个问题(引用自 AWS 托管的 Google NGrams 页面):

The sequence file key is the row number of the dataset stored as a LongWritable and the value is the raw data stored as TextWritable.

这意味着每一行都以额外的 Long + TAB 开头，因此您在映射器方法中执行的任何行解析也需要考虑前置信息。