我已经有一个非常耗时的Map Reduce作业在集群上运行。当我提交另一个作业时,它停留在以下位置,这表明它正在等待当前正在运行的作业完成:
hive> select distinct(circle) from vf_final_table_orc_format1;
Query ID = hduser_20181022153503_335ffd89-1528-49be-b091-21213d702a03
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 10
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1539782606189_0033, Tracking URL = http://secondary:8088/proxy/application_1539782606189_0033/
Kill Command = /home/hduser/hadoop/bin/hadoop job -kill job_1539782606189_0033
我目前正在对数据的
166GB
运行mapreduce作业。我的设置包括7 nodes
,其中5
是DN with 32GB RAM
和8.7TB HDD
,而1 NN
和1 SN
具有32 GB RAM
和1.1TB HDD
。我需要调整哪些设置才能并行执行作业?我目前正在使用
hadoop 2.5.2 version.
编辑:现在我的群集仅消耗8-10 GB RAM,每个节点32 GB。其他HIVE查询,MR Jobs卡住了,正在等待单个作业完成。如何增加内存消耗以促进更多并行执行的作业。这是
ps
命令的当前输出:[hduser@secondary ~]$ ps -ef | grep -i runjar | grep -v grep
hduser 110398 1 0 Nov11 ? 00:07:15 /opt/jdk1.8.0_77//bin/java -Dproc_jar -Xmx1000m
-Dhadoop.log.dir=/home/hduser/hadoop/logs -Dyarn.log.dir=/home/hduser/hadoop/logs
-Dhadoop.log.file=yarn.log -Dyarn.log.file=yarn.log -Dyarn.home.dir=
-Dyarn.id.str= -Dhadoop.root.logger=INFO,console -Dyarn.root.logger=INFO,console -Dyarn.policy.file=hadoop-policy.xml
-Dhadoop.log.dir=/home/hduser/hadoop/logs -Dyarn.log.dir=/home/hduser/hadoop/logs
-Dhadoop.log.file=yarn.log -Dyarn.log.file=yarn.log
-Dyarn.home.dir=/home/hduser/hadoop -Dhadoop.home.dir=/home/hduser/hadoop
-Dhadoop.root.logger=INFO,console
-Dyarn.root.logger=INFO,console
-classpath /home/hduser/hadoop/etc/hadoop:/home/hduser/hadoop/etc/hadoop:/home/hduser/hadoop/etc/hadoop:/home/hduser/hadoop/share/hadoop/common/lib/*:/home/hduser/hadoop/share/hadoop/common/*:/home/hduser/hadoop/share/hadoop/hdfs:/home/hduser/hadoop/share/hadoop/hdfs/lib/*:/home/hduser/hadoop/share/hadoop/hdfs/*:/home/hduser/hadoop/share/hadoop/yarn/lib/*:/home/hduser/hadoop/share/hadoop/yarn/*:/home/hduser/hadoop/share/hadoop/mapreduce/lib/*:/home/hduser/hadoop/share/hadoop/mapreduce/*:/home/hduser/hadoop/contrib/capacity-scheduler/*.jar:/home/hduser/hadoop/share/hadoop/yarn/*:/home/hduser/hadoop/share/hadoop/yarn/lib/*
org.apache.hadoop.util.RunJar abc.jar def.mydriver2 /raw_data /mr_output/
最佳答案
STEPS
Hive分阶段运行查询计划。
某些阶段取决于其他阶段,在之前的阶段完成之前无法启动。
但是,某些其他阶段可以与其他阶段同时运行。使阶段并行运行可以节省整体作业的运行时间。
要启用阶段的并行执行,请执行以下操作:
set hive.exec.parallel=true;
set hive.exec.parallel.thread.number=8;
并行执行将提高群集利用率。
如果集群的利用率已经很高,那么并行执行对整体性能将无济于事。
让我知道是否有帮助。
关于hadoop - Hadoop-如何在运行另一个mapreduce作业的同时运行它?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/52926999/