java - 运行 Hadoop 示例 Jar

标签 java macos hadoop bigdata

请帮忙。我在网上搜索,找不到任何东西。大多数类似的问题都没有答案或没有帮助。

您好,我正在尝试运行 Pi 的 Hadoop 示例。我的设置全部完成并成功。我运行了 bim/hadoop dfs -ls 并且没有收到任何错误。

但是这个:

Sanjanas-MacBook-Pro:hadoop sanjanaagarwal$ /usr/local/Hadoop/bin/hadoop jar $HADOOP_HOME/hadoop-examples-*.jar pi 10 100
Number of Maps  = 10
Samples per Map = 100
13/11/21 20:57:47 WARN hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/sanjanaagarwal/PiEstimator_TMP_3_141592654/in/part0 could only be replicated to 0 nodes, instead of 1
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1591)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:771)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1439)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1435)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:394)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1433)

    at org.apache.hadoop.ipc.Client.call(Client.java:1150)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
    at com.sun.proxy.$Proxy0.addBlock(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
    at com.sun.proxy.$Proxy0.addBlock(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3719)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3586)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2400(DFSClient.java:2792)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2987)

13/11/21 20:57:47 WARN hdfs.DFSClient: Error Recovery for block null bad datanode[0] nodes == null
13/11/21 20:57:47 WARN hdfs.DFSClient: Could not get block locations. Source file "/user/sanjanaagarwal/PiEstimator_TMP_3_141592654/in/part0" - Aborting...
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/sanjanaagarwal/PiEstimator_TMP_3_141592654/in/part0 could only be replicated to 0 nodes, instead of 1
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1591)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:771)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1439)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1435)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:394)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1433)

    at org.apache.hadoop.ipc.Client.call(Client.java:1150)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
    at com.sun.proxy.$Proxy0.addBlock(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
    at com.sun.proxy.$Proxy0.addBlock(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3719)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3586)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2400(DFSClient.java:2792)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2987)
13/11/21 20:57:47 ERROR hdfs.DFSClient: Exception closing file /user/sanjanaagarwal/PiEstimator_TMP_3_141592654/in/part0 : org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/sanjanaagarwal/PiEstimator_TMP_3_141592654/in/part0 could only be replicated to 0 nodes, instead of 1
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1591)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:771)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1439)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1435)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:394)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1433)

org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/sanjanaagarwal/PiEstimator_TMP_3_141592654/in/part0 could only be replicated to 0 nodes, instead of 1
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1591)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:771)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1439)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1435)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:394)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1433)

    at org.apache.hadoop.ipc.Client.call(Client.java:1150)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
    at com.sun.proxy.$Proxy0.addBlock(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
    at com.sun.proxy.$Proxy0.addBlock(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3719)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3586)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2400(DFSClient.java:2792)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2987)

我的 hadoop-env.sh 已准备就绪。仍然不知道这里有什么问题。如果有什么要检查的,请提及命令,因为我是 linux 的新手。谢谢

最佳答案

我认为您集群的 DataNode 没有运行。使用 jps 命令检查。

关于java - 运行 Hadoop 示例 Jar,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/20135642/

相关文章:

hadoop - 像Hive一样使用Presto插入覆盖目录

java - 我如何在 Kryo 中检测到 "End of File"?

java - 无法运行简单的java代码

用于解析使用 Pig 脚本生成的组的输出的 Java 程序

hadoop - Pig 在空集中生成结果

macos - .git/config 文件读取 "command not found"

java - 从 Map<List<String>, IInterface> 中的键列表中删除元素后获取键和值对

java - 只有一个事务数据源的 Spring @Transactional 行为

Swift - 将字典保存到 NSUserDefaults 时 macOS 程序卡住

macos - 升级到10.13 high-sierra后Meld无法启动