scala 连接 hbase 主机失败

Question

我写的 Scala 代码如下:

44 val config: Configuration = HBaseConfiguration.create()
 45     config.set("hbase.zookeeper.property.clientPort", zooKeeperClientPort)
 46     config.set("hbase.zookeeper.quorum", zooKeeperQuorum)
 47     config.set("zookeeper.znode.parent", zooKeeperZNodeParent)
 48     config.set("hbase.master", hbaseMaster)
 49     config.addResource("hbase-site.xml")
 50     config.addResource("hdfs-site.xml")
 51     HBaseAdmin.checkHBaseAvailable(config);
 52     val admin: HBaseAdmin = new HBaseAdmin(config)
 53     // descriptor.addColumn(new HColumnDescriptor(Bytes.toBytes("cfbfeature")))
 54     val conn = ConnectionFactory.createConnection(config)
 55     table = conn.getTable(TableName.valueOf(outputTable))

这是我的完整错误日志:

zooKeeperClientPort:2181, zooKeeperQuorum:zk1.hbase.busdev.usw2.cmcm.com,zk2.hbase.busdev.usw2.cmcm.com,zk3.hbase.busdev.usw2.cmcm.com,zooKeeperZNodeParent:/hbase, outputTable:RequestFeature, hbaseMaster:10.2.2.62:60000 16/12/13 08:25:56 WARN util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 16/12/13 08:25:56 WARN util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 16/12/13 08:25:56 WARN util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 16/12/13 08:25:57 INFO zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x6ae9e162 connecting to ZooKeeper ensemble=zk2.hbase.busdev.usw2.cmcm.com:2181,zk1.hbase.busdev.usw2.cmcm.com:2181,zk3.hbase.busdev.usw2.cmcm.com:2181 16/12/13 08:25:57 WARN util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 16/12/13 08:25:57 WARN util.DynamicClassLoader: Failed to identify the fs of dir hdfs://mycluster/hbase/lib, ignored java.net.UnknownHostException: unknown host: mycluster at org.apache.hadoop.ipc.Client$Connection.(Client.java:214) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1196) at org.apache.hadoop.ipc.Client.call(Client.java:1050) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225) at com.sun.proxy.$Proxy3.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379) at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119) at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:238) at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:203) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187) at org.apache.hadoop.hbase.util.DynamicClassLoader.(DynamicClassLoader.java:104) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.(ProtobufUtil.java:229) at org.apache.hadoop.hbase.ClusterId.parseFrom(ClusterId.java:64) at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:75) at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:86) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:833) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.(ConnectionManager.java:623) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238) at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218) at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119) at org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:2508) at com.cmcm.datahero.streaming.actor.ToHBaseActor.preStart(ToHBaseActor.scala:51) at akka.actor.Actor$class.aroundPreStart(Actor.scala:472) at com.cmcm.datahero.streaming.actor.ToHBaseActor.aroundPreStart(ToHBaseActor.scala:16) at akka.actor.ActorCell.create(ActorCell.scala:580) at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456) at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478) at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263) at akka.dispatch.Mailbox.run(Mailbox.scala:219) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 16/12/13 08:25:57 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x356c1ee7cac04c8

Answer 1

我终于把hbase和hdfs xml confiure放到了子路径src/main/resources。然后addResouce到hadoop configure。 但这不是我问题的核心。 hbase 包的 jar 版本应与 hbase 版本匹配。我修复了我的 build.sbt。下面贴出代码。希望可以帮助别人解决我遇到的错误。

libraryDependencies += "org.apache.hbase" % "hbase-client" % "1.0.0-cdh5.4.8"
libraryDependencies += "org.apache.hbase" % "hbase-common" % "1.0.0-cdh5.4.8"
libraryDependencies += "org.apache.hbase" % "hbase-server" % "1.0.0-cdh5.4.8"
libraryDependencies += "org.apache.hadoop" % "hadoop-core" % "2.6.0-mr1-cdh5.4.8"
libraryDependencies += "org.apache.hadoop" % "hadoop-hdfs" % "2.6.0-cdh5.4.8"
libraryDependencies += "org.apache.hadoop" % "hadoop-common" % "2.6.0-cdh5.5.4"
// libraryDependencies += "org.apache.hbase" % "hbase-client" % "1.0.0-CDH"
// libraryDependencies += "org.apache.hbase" % "hbase-common" % "1.0.0"
// libraryDependencies += "org.apache.hbase" % "hbase-server" % "1.0.0"

//scalaSource in Compile := baseDirectory.value / "src/main/scala"
//resourceDirectory in Compile := baseDirectory.value / "src/main/resources"
unmanagedBase := baseDirectory.value / "lib"
//unmanagedResourceDirectories in Compile += baseDirectory.value / "conf"
packAutoSettings
resolvers += Resolver.sonatypeRepo("snapshots")
resolvers += "cloudera repo" at "https://repository.cloudera.com/content/repositories/releases/"
resolvers += "cloudera repo1" at "https://repository.cloudera.com/artifactory/cloudera-repos/"