hadoop - 将hadoop集群升级到clodera 4 b1后，出现Invalid "mapreduce.jobtracker.address"配置错误

Question

我最近升级到 clodera4b1。在升级之前，mapreduce 作业运行良好，但现在当我执行任何 mapreduce 程序时，出现以下错误:

命令运行:

hadoop jar /usr/lib/hadoop/hadoop-mapreduce-examples-0.23.0-cdh4b1.jar grep *.xml /user/out/ 'dfs'


12/04/10 19:23:15 INFO mapreduce.Cluster: Failed to use org.apache.hadoop.mapred.LocalClientProtocolProvider due to error: Invalid "mapreduce.jobtracker.address" configuration value for LocalJobRunner : "dev-xxx:yyyy"
12/04/10 19:23:15 ERROR security.UserGroupInformation: PriviledgedActionException as:anchauhan (auth:SIMPLE) cause:java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
Exception in thread "main" java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
    at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:123)
    at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:85)
    at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:78)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1185)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1181)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1167)
    at org.apache.hadoop.mapreduce.Job.connect(Job.java:1180)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1209)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1233)
    at com.nextag.mapred.MerchantImport.doTask(MerchantImport.java:221)
    at com.nextag.mapred.Main.main(Main.java:9)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:200)

对于我的自定义 mapred 作业，我在编译 jar 文件时将库文件更改为新的，但没有成功

Answer 1

我们有同样的异常，原因是我们在 pom 文件中有错误的依赖项。我们指向用于 yarn 的 hadoop-client 版本 2.0.0-cdh4.0.0，但我们使用的是 MRv1，因此我们应该指向 2.0.0-mr1-cdh4.0.0。您的依赖项中不应包含任何 yarn jar 文件。

Reference