我想使用 HPROF 来分析我的 Hadoop 作业。问题是我得到了 TRACES
但 profile.out
文件中没有 CPU SAMPLES
。我在运行方法中使用的代码是:
/** Get configuration */
Configuration conf = getConf();
conf.set("textinputformat.record.delimiter","\n\n");
conf.setStrings("args", args);
/** JVM PROFILING */
conf.setBoolean("mapreduce.task.profile", true);
conf.set("mapreduce.task.profile.params", "-agentlib:hprof=cpu=samples," +
"heap=sites,depth=6,force=n,thread=y,verbose=n,file=%s");
conf.set("mapreduce.task.profile.maps", "0-2");
conf.set("mapreduce.task.profile.reduces", "");
/** Job configuration */
Job job = new Job(conf, "HadoopSearch");
job.setJarByClass(Search.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(NullWritable.class);
/** Set Mapper and Reducer, use identity reducer*/
job.setMapperClass(Map.class);
job.setReducerClass(Reducer.class);
/** Set input and output formats */
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
/** Set input and output path */
FileInputFormat.addInputPath(job, new Path("/user/niko/16M"));
FileOutputFormat.setOutputPath(job, new Path(cmd.getOptionValue("output")));
job.waitForCompletion(true);
return 0;
如何将 CPU SAMPLES
写入输出?
我在 stderr
上也有奇怪的错误消息,但我认为它不相关,因为当分析设置为 false 或启用分析的代码被注释掉时它也会出现。错误是
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.impl.MetricsSystemImpl).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
最佳答案
Yarn(或 MRv1)在您的工作完成后立即终止容器。 不能将 CPU 样本写入您的分析文件。事实上,您的踪迹也应该被截断。
您必须添加以下选项(或您的 Hadoop 版本上的等效选项):
yarn.nodemanager.sleep-delay-before-sigkill.ms = 30000
# No. of ms to wait between sending a SIGTERM and SIGKILL to a container
yarn.nodemanager.process-kill-wait.ms = 30000
# Max time to wait for a process to come up when trying to cleanup a container
mapreduce.tasktracker.tasks.sleeptimebeforesigkill = 30000
# Same en MRv1 ?
(30 秒似乎足够了)
关于java - Hadoop HPROF 分析未写入 CPU 样本,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/25983999/