hadoop - 无法将现有文件附加到 HDFS

Question

我在 VM 上运行单节点 Hadoop 1.2.1 集群。

我的 hdfs-site.xml 如下所示:

<configuration>
<property>
  <name>dfs.replication</name>
  <value>1</value>
  <description>Default block replication.
 </description>
</property>
<property>
  <name>dfs.support.append</name>
  <value>true</value>
  <description>Does HDFS allow appends to files?
  </description>
</property>
</configuration>

现在，当我尝试从 Eclipse 运行以下代码时，它始终返回 false:

Configuration config = new Configuration();
    config.set("mapred.job.tracker","10.0.0.6:54311");
    config.set("fs.default.name","hdfs://10.0.0.6:54310");
    FileSystem fs = FileSystem.get(config);
    boolean flag = Boolean.getBoolean(fs.getConf().get("dfs.support.append"));
    System.out.println("dfs.support.append is set to be " + flag);

现在，如果我尝试附加到现有文件，我将收到以下错误:

org.apache.hadoop.ipc.RemoteException: java.io.IOException: Append is not supported. Please see the dfs.support.append configuration parameter
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1781)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.append(NameNode.java:725)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426)

    at org.apache.hadoop.ipc.Client.call(Client.java:1113)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
    at com.sun.proxy.$Proxy1.append(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
    at com.sun.proxy.$Proxy1.append(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:933)
    at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:922)
    at org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSystem.java:196)
    at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:659)
    at com.vanilla.hadoop.AppendToHdfsFile.main(AppendToHdfsFile.java:29)

怎么了？我错过了什么吗？

Answer 1

您应该尝试使用 2.X.X 版本或 0.2X 版本，因为在 hadoop 0.20.2 之后在 hdfs 上附加文件。查看更多信息 here和 here