scala - 控制配置设置 Apache Spark UTF 编码以写入为 saveAsTextFile

Question

那么在使用 saveAsTextFile(path) 时如何告诉 Spark 使用哪种 UTF？当然，如果知道所有字符串都是 UTF-8 那么它将节省 2 倍的磁盘空间! (假设像java一样默认UTF是16)

Answer 1

saveAsTextFile实际上使用Text来自hadoop，编码为UTF-8。

def saveAsTextFile(path: String, codec: Class[_ <: CompressionCodec]) {
    this.map(x => (NullWritable.get(), new Text(x.toString)))
      .saveAsHadoopFile[TextOutputFormat[NullWritable, Text]](path, codec)
  }

来自 Text.java:

public class Text extends BinaryComparable
    implements WritableComparable<BinaryComparable> {

  static final int SHORT_STRING_MAX = 1024 * 1024;

  private static ThreadLocal<CharsetEncoder> ENCODER_FACTORY =
    new ThreadLocal<CharsetEncoder>() {
      protected CharsetEncoder initialValue() {
        return Charset.forName("UTF-8").newEncoder().
               onMalformedInput(CodingErrorAction.REPORT).
               onUnmappableCharacter(CodingErrorAction.REPORT);
    }
  };

  private static ThreadLocal<CharsetDecoder> DECODER_FACTORY =
    new ThreadLocal<CharsetDecoder>() {
    protected CharsetDecoder initialValue() {
      return Charset.forName("UTF-8").newDecoder().
             onMalformedInput(CodingErrorAction.REPORT).
             onUnmappableCharacter(CodingErrorAction.REPORT);
    }
  };

如果你想保存为 UTF-16 我想你可以使用 saveAsHadoopFile与 org.apache.hadoop.io.BytesWritable并获取 java String 的字节(正如你所说，这将是 UTF-16)。像这样的事情:
saveAsHadoopFile[SequenceFileOutputFormat[NullWritable, BytesWritable]](path)
您可以从 "...".getBytes("UTF-16") 获取字节