java - 具有独立 hbase 的 spring hbase 示例

Question

在我的本地开发 Ubuntu 14 机器上，我无法让 hadoop 和 hbase 独立运行。

但是可以让 hbase 直接使用文件系统(通过创建表并将值放入 hbase shell 进行验证)。

现在我想从 spring 读取这张表。我从 https://github.com/spring-projects/spring-hadoop-samples/tree/master/hbase 的代码开始

如果我更改 qa 服务器的 IP 和端口，它就可以工作。但如果我想连接到本地 hbase 却无法连接。

hbase-站点

   <property>
      <name>hbase.rootdir</name>
      <!-- value>hdfs://localhost:50070/hbase</value --> <!-- 8030-->
    <value>file:///usr/local/hbase/HFiles3</value>
   </property>

   <property>
    <name>hbase.zookeeper.property.dataDir</name>
    <value>/usr/local/hbase/zookeeper3</value>
  </property>     

   <property>
    <name>hbase.master.info.port</name>
    <value>8095</value>
  </property>

   <property>
    <name>hbase.regionserver.info.port</name>
    <value>8096</value> <!-- changed as i dont know where it was connecting other wise none of the usual ports -->
  </property>

应用程序上下文.xml

    <context:property-placeholder location="hbase.properties"/>

    <context:component-scan base-package="org.springframework.samples.hadoop.hbase"/>

    <hdp:configuration id="hadoopConfiguration">
      fs.defaultFS=file:///usr/local/hbase/HFiles3
    </hdp:configuration>

<!--    configuration-ref="hadoopConfiguration" -->
    <hdp:hbase-configuration configuration-ref="hadoopConfiguration" zk-quorum="${hbase.zk.host}" zk-port="${hbase.zk.port}"/>

    <bean id="hbaseTemplate" class="org.springframework.data.hadoop.hbase.HbaseTemplate">
        <property name="configuration" ref="hbaseConfiguration"/>
    </bean>

hbase.properties

hbase.zk.host=localhost
hbase.zk.port=8095

日志(部分): ...

12:20:21,763  INFO           org.apache.zookeeper.ZooKeeper: 100 - Client environment:user.home=/home/t
12:20:21,763  INFO           org.apache.zookeeper.ZooKeeper: 100 - Client environment:user.dir=/home/t/prog/r/github/spring-hadoop-samples/hbase
12:20:21,765  INFO           org.apache.zookeeper.ZooKeeper: 438 - Initiating client connection, connectString=localhost:8095 sessionTimeout=90000 watcher=hconnection-0x5a7fd55c, quorum=localhost:8095, baseZNode=/hbase
12:20:21,802  INFO oop.hbase.zookeeper.RecoverableZooKeeper: 120 - Process identifier=hconnection-0x5a7fd55c connecting to ZooKeeper ensemble=localhost:8095
12:20:21,805  INFO          org.apache.zookeeper.ClientCnxn: 975 - Opening socket connection to server t/127.0.0.1:8095. Will not attempt to authenticate using SASL (unknown error)
12:20:21,828  INFO          org.apache.zookeeper.ClientCnxn: 852 - Socket connection established to t/127.0.0.1:8095, initiating session
12:20:21,834  WARN          org.apache.zookeeper.ClientCnxn:1102 - Session 0x0 for server t/127.0.0.1:8095, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Packet len1213486160 is out of range!
    at org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:112)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:79)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
12:20:21,947  WARN oop.hbase.zookeeper.RecoverableZooKeeper: 253 - Possibly transient ZooKeeper, quorum=localhost:8095, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
12:20:21,947  INFO rg.apache.hadoop.hbase.util.RetryCounter: 155 - Sleeping 1000ms before retry #0...
12:20:23,325  INFO          org.apache.zookeeper.ClientCnxn: 975 - Opening socket connection to server t/127.0.0.1:8095. Will not attempt to authenticate using SASL (unknown error)
12:20:23,326  INFO          org.apache.zookeeper.ClientCnxn: 852 - Socket connection established to t/127.0.0.1:8095, initiating session
12:20:23,327  WARN          org.apache.zookeeper.ClientCnxn:1102 - Session 0x0 for server t/127.0.0.1:8095, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Packet len1213486160 is out of range!

...

我可以在没有hadoop的情况下为本地开发运行spring hbase吗？我是否需要运行独立的 Zookeeper 并将该端口指定为 hbase.zk.port ？ hbase.zk.port 是什么？

Answer 1

在独立模式下，它也连接到 2181。从hbase的日志来看:

2015-08-01 12:30:36,489 INFO [main] server.ZooKeeperServer: Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /usr/local/hbase/zookeeper3/zookeeper_0/version-2 snapdir /usr/local/hbase/zookeeper3/zookeeper_0/version-2 2015-08-01 12:30:36,504 INFO [main] server.NIOServerCnxnFactory: binding to port 0.0.0.0/0.0.0.0:2181 2015-08-01 12:30:36,638 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /127.0.0.1:60489

示例有效 - 'hadoop' 本地文件 URL 和带有默认 2181 端口的 zk

<hdp:configuration id="hadoopConfiguration">
      fs.defaultFS=file:///usr/local/hbase/HFiles3
    </hdp:configuration> 

hbase.zk.port=2181