我对运行hadoop mapreduce工作有疑问。我有一个工作人员,按加入日期划分。
创建这样的语句:
创建表人员(id int,age int),并按(join_date字符串)行格式分隔的字段以'\;'结尾;
我将一些数据分配给分区“20130921”,然后在执行下面的语句时,结果正常:
select count(*) from staff where join_date='20130921';**
但是,当我在分区“20130922”(没有数据的分区)上执行时,map reduce作业挂起的时间太长,似乎永远运行:
hive> select count(*) from staff where join_date='20130922';**
Total MapReduce jobs = 1**
Launching Job 1 out of 1**
**Number of reduce tasks determined at compile time: 1**
**In order to change the average load for a reducer (in bytes):**
set hive.exec.reducers.bytes.per.reducer=<number>**
**In order to limit the maximum number of reducers:**
set hive.exec.reducers.max=<number>**
**In order to set a constant number of reducers:**
set mapred.reduce.tasks=<number>**
**Starting Job** = `job_201309231116_0131, Tracking URL = ....jobid=job_201309231116_0131`
**Kill Command** = `/u01/hadoop-0.20.203.0/bin/../bin/hadoop job -kill job_201309231116_0131`
Hadoop job information for Stage-1: number of mappers: 0; number of reducers: 1
2013-09-23 17:19:07,182 Stage-1 map = 0%, reduce = 0%
2013-09-23 17:19:07,182 Stage-1 map = 0%, reduce = 0%
2013-09-23 17:19:07,182 Stage-1 map = 0%, reduce = 0%
jobtracker显示reduce任务待处理,而这项工作似乎无法完成。
我正在使用hadoop-0.20.203.0和hive-0.10.0。我整天用Google搜索,但没有发现任何主题有相同的问题,请帮助我。
最好的祝福。
最佳答案
您的Hive安装似乎有问题。我遇到了类似的问题。您可以尝试重新启动Hive Server和Hive Metastore。这解决了我的问题。
关于hadoop - hadoop map减少待处理的作业太长时间,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/18962938/