hadoop - hbase shell缺少类名('org.apache.log4j.level')

Question

尝试启动hbase 2.2.0 shell时出现以下错误。
我已经下载了hbase-2.2.0 binary tar.gz并解压缩到Windows x64环境16GB RAM中。

如以下错误消息所示，由于未找到Line 81类，它是由%HBASE_HOME%\bin\hirb.rb的org.apache.log4j.Level引起的

我在%HBASE_HOME%\conf\hbase-env.cmd中设置了HBase环境变量，并将所有%HBASE_HOME%\lib jar 添加到HBASE_CLASSPATH中(如下所示)，但是得到了相同的错误

%HBASE_HOME%\conf\hbase-env.cmd

set JAVA_HOME=c:\Program Files\Java\jdk1.8.0_101


set HBASE_CLASSPATH=%HBASE_HOME%\lib;%HBASE_HOME%\lib\client-facing-thirdparty;%HBASE_HOME%\lib\ruby;%HBASE_HOME%\lib\zkcli;%HBASE_HOME%\lib\shaded-clients

set HBASE_HEAPSIZE=8000

set HBASE_OPTS="-XX:+UseConcMarkSweepGC" "-Djava.net.preferIPv4Stack=true"

set SERVER_GC_OPTS="-verbose:gc" "-XX:+PrintGCDetails" "-XX:+PrintGCDateStamps" %HBASE_GC_OPTS%

set CLIENT_GC_OPTS="-verbose:gc" "-XX:+PrintGCDetails" "-XX:+PrintGCDateStamps" %HBASE_GC_OPTS%

set HBASE_USE_GC_LOGFILE=true

set HBASE_JMX_BASE="-Dcom.sun.management.jmxremote.ssl=false" "-Dcom.sun.management.jmxremote.authenticate=false"
set HBASE_MASTER_OPTS=%HBASE_JMX_BASE% "-Dcom.sun.management.jmxremote.port=10101"
set HBASE_REGIONSERVER_OPTS=%HBASE_JMX_BASE% "-Dcom.sun.management.jmxremote.port=10102"
set HBASE_THRIFT_OPTS=%HBASE_JMX_BASE% "-Dcom.sun.management.jmxremote.port=10103"
set HBASE_ZOOKEEPER_OPTS=%HBASE_JMX_BASE% -Dcom.sun.management.jmxremote.port=10104"


set HBASE_REGIONSERVERS=%HBASE_HOME%\conf\regionservers

set HBASE_IDENT_STRING=%USERNAME%

set HBASE_MANAGES_ZK=true

自定义HBase和Zookeeper数据目录(与HBASE_HOME不同)，并将分布式模式指定为false，以独立模式运行。

%HBASE_HOME%\conf\hbase-site.xml

<configuration>
<property>
    <name>hbase.rootdir</name>
    <value>file:///D:/HBase/data</value>
</property>
<property>
    <name>hbase.zookeeper.property.dataDir</name>
    <value>file:///D:/HBase/zookeeper</value>
</property>
<property>
    <name>hbase.unsafe.stream.capability.enforce</name>
    <value>false</value>
    <description>
      Controls whether HBase will check for stream capabilities (hflush/hsync).

      Disable this if you intend to run on LocalFileSystem, denoted by a rootdir
      with the 'file://' scheme, but be mindful of the NOTE below.

      WARNING: Setting this to false blinds you to potential data loss and
      inconsistent system state in the event of process and/or node failures. If
      HBase is complaining of an inability to use hsync or hflush it's most
      likely not a false positive.
    </description>
</property>
<property>
    <name>hbase.cluster.distributed</name>
    <value>false</value>
</property>
</configuration>

下面是hirb.rb脚本的第81行

Line 81: log_level = org.apache.log4j.Level::ERROR

错误:

NameError: missing class name (`org.apache.log4j.Level')
  method_missing at org/jruby/javasupport/JavaPackage.java:259
  <main> at D:\HBase\hbase-2.2.0-bin.tar\hbase-2.2.0-bin\hbase-2.2.0\bin\hirb.rb:81

Console log

D:\HBase\hbase-2.2.0-bin.tar\hbase-2.2.0-bin\hbase-2.2.0>bin\hbase shell
2019-09-23T17:54:25.354+0530: [GC (Allocation Failure) 2019-09-23T17:54:25.354+0530: [ParNew: 69952K->8703K(78656K), 0.0227532 secs] 69952K->15376K(253440K), 0.0232659 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
2019-09-23T17:54:25.377+0530: [GC (CMS Initial Mark) [1 CMS-initial-mark: 6672K(174784K)] 15401K(253440K), 0.0024235 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2019-09-23T17:54:25.380+0530: [CMS-concurrent-mark-start]
2019-09-23T17:54:25.504+0530: [CMS-concurrent-mark: 0.124/0.124 secs] [Times: user=0.42 sys=0.02, real=0.12 secs]
2019-09-23T17:54:25.505+0530: [CMS-concurrent-preclean-start]
2019-09-23T17:54:25.507+0530: [CMS-concurrent-preclean: 0.002/0.002 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2019-09-23T17:54:25.508+0530: [CMS-concurrent-abortable-preclean-start]
2019-09-23T17:54:26.668+0530: [CMS-concurrent-abortable-preclean: 0.694/1.160 secs] [Times: user=3.84 sys=0.22, real=1.16 secs]
2019-09-23T17:54:26.669+0530: [GC (CMS Final Remark) [YG occupancy: 49337 K (78656 K)]2019-09-23T17:54:26.669+0530: [Rescan (parallel) , 0.0086939 secs]2019-09-23T17:54:26.678+0530: [weak refs processing, 0.0001806 secs]2019-09-23T17:54:26.678+0530: [class unloading, 0.0045989 secs]2019-09-23T17:54:26.683+0530: [scrub symbol table, 0.0034678 secs]2019-09-23T17:54:26.686+0530: [scrub string table, 0.0008139 secs][1 CMS-remark: 6672K(174784K)] 56010K(253440K), 0.0193630 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
2019-09-23T17:54:26.693+0530: [CMS-concurrent-sweep-start]
2019-09-23T17:54:26.694+0530: [CMS-concurrent-sweep: 0.001/0.001 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2019-09-23T17:54:26.694+0530: [CMS-concurrent-reset-start]
2019-09-23T17:54:26.704+0530: [CMS-concurrent-reset: 0.009/0.010 secs] [Times: user=0.05 sys=0.00, real=0.01 secs]
2019-09-23T17:54:26.941+0530: [GC (Allocation Failure) 2019-09-23T17:54:26.941+0530: [ParNew: 78655K->8704K(78656K), 0.0437466 secs] 85328K->26040K(253440K), 0.0442171 secs] [Times: user=0.08 sys=0.02, real=0.04 secs]
2019-09-23T17:54:27.150+0530: [GC (Allocation Failure) 2019-09-23T17:54:27.150+0530: [ParNew: 78656K->8704K(78656K), 0.0176586 secs] 95992K->31897K(253440K), 0.0188060 secs] [Times: user=0.05 sys=0.00, real=0.02 secs]
2019-09-23T17:54:27.408+0530: [GC (Allocation Failure) 2019-09-23T17:54:27.408+0530: [ParNew: 78656K->8548K(78656K), 0.0068995 secs] 101849K->32196K(253440K), 0.0081199 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2019-09-23T17:54:27.649+0530: [GC (Allocation Failure) 2019-09-23T17:54:27.650+0530: [ParNew: 78500K->8677K(78656K), 0.0069747 secs] 102148K->32515K(253440K), 0.0081496 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2019-09-23T17:54:27.807+0530: [GC (Allocation Failure) 2019-09-23T17:54:27.807+0530: [ParNew: 78629K->7285K(78656K), 0.0074532 secs] 102467K->32051K(253440K), 0.0085688 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
...
2019-09-23T17:54:44.632+0530: [GC (Allocation Failure) 2019-09-23T17:54:44.633+0530: [ParNew: 77107K->8704K(78656K), 0.0066891 secs] 159919K->91658K(253440K), 0.0080000 secs] [Times: user=0.06 sys=0.00, real=0.01 secs]
2019-09-23T17:54:45.217+0530: [GC (Allocation Failure) 2019-09-23T17:54:45.218+0530: [ParNew: 78656K->8704K(78656K), 0.0125041 secs] 161610K->93529K(253440K), 0.0137068 secs] [Times: user=0.06 sys=0.00, real=0.01 secs]
NameError: missing class name (`org.apache.log4j.Level')
  method_missing at org/jruby/javasupport/JavaPackage.java:259
          <main> at D:\HBase\hbase-2.2.0-bin.tar\hbase-2.2.0-bin\hbase-2.2.0\bin\hirb.rb:81
Heap
 par new generation   total 78656K, used 32489K [0x00000005cc000000, 0x00000005d1550000, 0x00000005e0cc0000)
  eden space 69952K,  34% used [0x00000005cc000000, 0x00000005cd73a7e0, 0x00000005d0450000)
  from space 8704K, 100% used [0x00000005d0450000, 0x00000005d0cd0000, 0x00000005d0cd0000)
  to   space 8704K,   0% used [0x00000005d0cd0000, 0x00000005d0cd0000, 0x00000005d1550000)
 concurrent mark-sweep generation total 174784K, used 84825K [0x00000005e0cc0000, 0x00000005eb770000, 0x00000007c0000000)
 Metaspace       used 35739K, capacity 36202K, committed 36360K, reserved 1079296K
  class space    used 6361K, capacity 6500K, committed 6524K, reserved 1048576K

运行start-hbase会导致以下错误:

D:\HBase\hbase-2.2.0-bin.tar\hbase-2.2.0-bin\hbase-2.2.0>bin\start-hbase
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory
        at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:187)
        at org.apache.hadoop.hbase.util.HBaseConfTool.main(HBaseConfTool.java:39)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 2 more
ERROR: Could not determine the startup mode.

Answer 1

我已经解决了这个问题。

在%HBASE_HOME%\conf\hbase-env.cmd中进行了以下环境变量设置

如果先设置了JAVA_HOME，则本例中的JAVA_HOME=C:\Program Files\Java\jdk1.8.0_101在Windows环境变量中设置

        set JAVA_HOME=%JAVA_HOME% 
        set HBASE_CLASSPATH=%HBASE_HOME%\lib\client-facing-thirdparty\*
        set HBASE_HEAPSIZE=8000
        set HBASE_OPTS="-XX:+UseConcMarkSweepGC" "-Djava.net.preferIPv4Stack=true"
        set SERVER_GC_OPTS="-verbose:gc" "-XX:+PrintGCDetails" "-XX:+PrintGCDateStamps" %HBASE_GC_OPTS%
        set HBASE_USE_GC_LOGFILE=true
        set HBASE_JMX_BASE="-Dcom.sun.management.jmxremote.ssl=false" "-Dcom.sun.management.jmxremote.authenticate=false"
        set HBASE_MASTER_OPTS=%HBASE_JMX_BASE% "-Dcom.sun.management.jmxremote.port=10101"
        set HBASE_REGIONSERVER_OPTS=%HBASE_JMX_BASE% "-Dcom.sun.management.jmxremote.port=10102"
        set HBASE_THRIFT_OPTS=%HBASE_JMX_BASE% "-Dcom.sun.management.jmxremote.port=10103"
        set HBASE_ZOOKEEPER_OPTS=%HBASE_JMX_BASE% -Dcom.sun.management.jmxremote.port=10104"
        set HBASE_REGIONSERVERS=%HBASE_HOME%\conf\regionservers
        set HBASE_LOG_DIR=%HBASE_HOME%\logs
        set HBASE_IDENT_STRING=%USERNAME%
        set HBASE_MANAGES_ZK=true

由于以独立模式运行，因此仅在%HBASE_HOME%\conf\hbase-site.xml中添加以下属性

IMP:无需添加hbase.rootdir或hbase.zookeeper.property.dataDir

<configuration>
        <property>
            <name>hbase.cluster.distributed</name>
            <value>false</value>
        </property>
</configuration>