hadoop - 吉拉夫ZooKeeper端口问题

标签 hadoop apache-zookeeper hortonworks-data-platform giraph

我正在尝试运行Giraph Quick Start中描述的SimpleShortestPathsVertex(又名SimpleShortestPathComputation)示例。我正在使用VirtualBox在Hortonworks沙盒实例(HDP 2.1)上运行此程序,并使用配置文件hadoop_2.0.0打包了giraph.jar。

当我尝试使用运行示例

hadoop jar giraph.jar org.apache.giraph.GiraphRunner 
org.apache.giraph.examples.SimpleShortestPathsVertex -vif 
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip 
/user/hue/tinygraph.txt -of org.apache.giraph.io.formats.IdWithValueTextOutputFormat 
-op /user/hue/output/shortestpaths -w 1

我得到以下异常
2014-04-30 07:22:15,390 INFO [main] org.apache.giraph.zk.ZooKeeperManager: onlineZooKeeperServers: Connect attempt 0 of 10 max trying to connect to sandbox.hortonworks.com:22181 with poll msecs = 3000
2014-04-30 07:22:15,396 WARN [main] org.apache.giraph.zk.ZooKeeperManager: onlineZooKeeperServers: Got ConnectException
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:701)
at org.apache.giraph.graph.GraphTaskManager.startZooKeeperManager(GraphTaskManager.java:357)
at org.apache.giraph.graph.GraphTaskManager.setup(GraphTaskManager.java:188)
at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:60)
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:90)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

我发现了一个解决方法-看来Giraph希望ZooKeeper在端口22181上运行,而实际上它在2181上运行。我只是使用Ambari界面将ZooKeeper设置为在22181上运行(转到http://127.0.0.1:8080/,登录admin /管理员,服务选项卡,ZooKeeper,并将端口更改为22181,保存并执行服务操作->全部重新启动。

有没有人对此问题有更好的解决方案?是否有通过其指定端口的配置,或者在Giraph源代码中此端口是错字?

最佳答案

是的,您可以使用-Dgiraph.zkList=localhost:2181选项指定每次运行Giraph作业的时间

另外,您可以在Hadoop配置中对其进行设置,然后不必在每次提交Giraph作业时都传递此选项。为此,在conf/core-site.xml文件中添加以下行:
<property><name>giraph.zkList</name><value>localhost:2181</value></property>
[请检查语法,我不记得它的语法,目前我无法访问群集进行检查]

关于hadoop - 吉拉夫ZooKeeper端口问题,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/23393259/

相关文章:

hadoop - 启动后如何在沙箱上启动h2o

java - 使用 Java 将目录内容从本地文件系统移动到 HDFS

linux - 如何在 linux 中访问 hadoop web UI?

scala - 我想将 Hive 中所有现有的 UDTF 转换为 Scala 函数并从 Spark SQL 使用它

java - 在 Zookeeper 节点上调用 getChildren 时忽略锁定节点

hadoop - 配置单元多次插入使用DISTINCT select语句出错

java - 如何获取zookeeper中路径的创建模式

java - Zookeeper 3.5.x 向后兼容 Zookeeper 3.4.x 客户端

hadoop - Hive/Tez作业无法开始