hadoop - 在两节点 hadoop 集群中运行映射缩减作业时出错 : Too many fetch-failures, 读取任务输出时出错 http://localhost:50060/tasklog

Question

我已经建立了一个双节点 hadoop 集群。我启动了 hadoop 文件系统和 map reduced daemons 没有错误，并验证它们正在主从上运行。我可以使用命令 bin/hadoop dfs -getmerge hdfs://my.domain.com:54310/user/wordcount/sunzi.txt/tmp/wordcount 从主节点和从节点读取输入文件。当我运行 map reduce 作业时，我在输出中看到错误。作业最终完成，但 reduce 部分花费了很长时间，并且每次打印错误时它都会继续返回 map 任务。我的站点配置文件引用了 master 的 dns 名称，所以我不知道为什么作业要尝试从“localhost”读取任务输出

12/12/20 10:47:36 INFO input.FileInputFormat: Total input paths to process : 7
12/12/20 10:47:36 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/12/20 10:47:36 WARN snappy.LoadSnappy: Snappy native library not loaded
12/12/20 10:47:36 INFO mapred.JobClient: Running job: job_201212201046_0001
12/12/20 10:47:37 INFO mapred.JobClient:  map 0% reduce 0%
12/12/20 10:47:44 INFO mapred.JobClient:  map 42% reduce 0%
12/12/20 10:47:45 INFO mapred.JobClient:  map 57% reduce 0%
12/12/20 10:47:49 INFO mapred.JobClient:  map 71% reduce 0%
12/12/20 10:47:50 INFO mapred.JobClient:  map 100% reduce 0%
12/12/20 10:47:54 INFO mapred.JobClient:  map 100% reduce 2%
12/12/20 10:48:08 INFO mapred.JobClient: Task Id : attempt_201212201046_0001_m_000002_0, Status : FAILED
Too many fetch-failures
12/12/20 10:48:08 WARN mapred.JobClient: Error reading task outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_201212201046_0001_m_000002_0&filter=stdout
12/12/20 10:48:08 WARN mapred.JobClient: Error reading task outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_201212201046_0001_m_000002_0&filter=stderr
12/12/20 10:48:08 INFO mapred.JobClient: Task Id : attempt_201212201046_0001_m_000001_0, Status : FAILED
Too many fetch-failures
12/12/20 10:48:09 INFO mapred.JobClient:  map 85% reduce 2%
12/12/20 10:48:10 INFO mapred.JobClient:  map 71% reduce 2%
12/12/20 10:48:11 INFO mapred.JobClient:  map 85% reduce 2%
12/12/20 10:48:12 INFO mapred.JobClient:  map 100% reduce 2%
12/12/20 10:48:33 INFO mapred.JobClient:  map 100% reduce 3%
12/12/20 10:48:34 INFO mapred.JobClient:  map 100% reduce 4%
12/12/20 10:48:47 INFO mapred.JobClient: Task Id : attempt_201212201046_0001_m_000003_0, Status : FAILED
Too many fetch-failures

我在任务跟踪器日志中看到了这个:

2012-12-20 10:51:22,255 WARN org.apache.hadoop.mapred.TaskTracker: Unknown child with bad map output: attempt_201212201046_0001_m_000005_2. Ignored.
2012-12-20 10:51:22,256 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.0.1:50060, dest: 127.0.0.1:49774, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201212201046_0001_m_000005_2, duration: 1870835
2012-12-20 10:51:22,257 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/apollo/jobcache/job_201212201046_0001/attempt_201212201046_0001_m_000005_2/output/file.out.index in any of the configured local directories
2012-12-20 10:51:23,225 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201212201046_0001_r_000002_0 0.19047621% reduce > copy (4 of 7 at 0.00 MB/s) > 
2012-12-20 10:51:26,239 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201212201046_0001_r_000002_0 0.19047621% reduce > copy (4 of 7 at 0.00 MB/s) > 
2012-12-20 10:51:26,372 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201212201046_0001_r_000000_0 0.19047621% reduce > copy (4 of 7 at 0.00 MB/s) > 
2012-12-20 10:51:32,255 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201212201046_0001_r_000002_0 0.19047621% reduce > copy (4 of 7 at 0.00 MB/s) > 
2012-12-20 10:51:32,387 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201212201046_0001_r_000000_0 0.19047621% reduce > copy (4 of 7 at 0.00 MB/s) > 
2012-12-20 10:51:35,401 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201212201046_0001_r_000000_0 0.19047621% reduce > copy (4 of 7 at 0.00 MB/s) > 
2012-12-20 10:51:37,116 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201212201046_0001_m_000005_2,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/apollo/jobcache/job_201212201046_0001/attempt_201212201046_0001_m_000005_2/output/file.out.index in any of the configured local directories
        at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429)
        at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
        at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:4009)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
        at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:848)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

Answer 1

对于 future 的 googlers，我能够通过从我的/etc/hosts 文件中删除以下条目来解决这个问题

127.0.0.1 localhost

虽然我不明白为什么 Hadoop 会尝试使用本地主机。