hadoop - 无法识别的 Hadoop 主要版本号 : 3. 0.0-beta1 在 org.apache.hadoop.hive.shims.ShimLoader.getMajorVersion

Question

我正在为安装在 Windows 10 环境中的 hadoop 3.0 beta1 使用 Windows native 库。 HDFS 和 Yarn 成功启动。安装 Hive 2.2 与安装了 derby db 的相同 Windows native 库。但在运行 hive 服务器时失败。

以下是运行 Hive 时收到的错误。
Hive 与 hadoop 3.0 beta 1 是否有任何版本兼容性，这将解决以下错误。

2017-10-22 12:25:53,002 ERROR shims.ShimLoader: Error loading shims
java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.0.0-beta1
        at org.apache.hadoop.hive.shims.ShimLoader.getMajorVersion(ShimLoader.java:177)
        at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:144)
        at org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:99)
        at org.apache.hadoop.hive.ql.udf.UDAFPercentile.<clinit>(UDAFPercentile.java:51)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hive.common.util.ReflectionUtil.newInstance(ReflectionUtil.java:83)
        at org.apache.hadoop.hive.ql.exec.Registry.registerUDAF(Registry.java:238)
        at org.apache.hadoop.hive.ql.exec.Registry.registerUDAF(Registry.java:231)
        at org.apache.hadoop.hive.ql.exec.FunctionRegistry.<clinit>(FunctionRegistry.java:430)
        at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:243)
        at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:227)
        at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:384)
        at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:328)
        at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:308)
        at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:284)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:581)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:544)
        at org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:127)
        at org.apache.hive.service.cli.CLIService.init(CLIService.java:112)
        at org.apache.hive.service.cli.thrift.EmbeddedThriftBinaryCLIService.init(EmbeddedThriftBinaryCLIService.java:45)
        at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:165)
        at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:208)
        at org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:145)
        at org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:209)
        at org.apache.hive.beeline.Commands.connect(Commands.java:1497)
        at org.apache.hive.beeline.Commands.connect(Commands.java:1392)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:54)
        at org.apache.hive.beeline.BeeLine.execCommandWithPrefix(BeeLine.java:1105)
        at org.apache.hive.beeline.BeeLine.embeddedConnect(BeeLine.java:909)
        at org.apache.hive.beeline.BeeLine.defaultConnect(BeeLine.java:926)
        at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:872)
        at org.apache.hive.beeline.cli.HiveCli.runWithArgs(HiveCli.java:35)
        at org.apache.hive.beeline.cli.HiveCli.main(HiveCli.java:29)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
java.lang.ExceptionInInitializerError
Beeline version 2.2.0 by Apache Hive
2017-10-22 12:25:53,072 WARN conf.HiveConf: HiveConf of name hive.server2.enable.impersonation does not exist
2017-10-22 12:25:53,121 WARN conf.HiveConf: HiveConf of name hive.server2.enable.impersonation does not exist

Answer 1

编辑 : 截至 2018 年 5 月 21 日，Hive 3.0.0 发布并支持 Hadoop 3。请参阅 JIRA Changelog更多细节。

在提出这个问题时，Hive 尚未与 Hadoop 3 集成。这些是与您的问题相关的错误。

https://issues.apache.org/jira/browse/HIVE-15016 - 使用 Hadoop 3.0.0-beta1 运行测试
https://issues.apache.org/jira/browse/HIVE-15326 - Hive shims 报告无法识别的 Hadoop 主要版本号:3.0.0-alpha2-SNAPSHOT

和 downloads page显示:



这意味着仍然不支持与 Hadoop 3 的集成。