我正在尝试设置 oozie 和 sqoop 工作流(我想将 mySql 数据备份到我的 hdfs 中)。 但是当我尝试开始我的工作时,我遇到了困难。
我正在使用 hadoop2(工作中的 hdfs 节点),oozie 的最新版本。 我在我的电脑上安装了 oozie 服务器(我想在部署它之前测试它),在 oozie conf/haddop-conf 目录)和我在服务器上的 hdfs。
我做了一个基本的工作流程(测试目的,我只是想看看sqoop是否工作)是这样的:
<workflow-app name="Sqoop" xmlns="uri:oozie:workflow:0.4">
<start to="Sqoop"/>
<action name="Sqoop">
<sqoop xmlns="uri:oozie:sqoop-action:0.2">
<job-tracker>yarn.resourcemanager.address:8040</job-tracker>
<name-node>hdfs://hdfs-server:54310</name-node>
<command>job --list</command>
</sqoop>
<ok to="end"/>
<error to="kill"/>
</action>
<kill name="kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app>
我将此工作流放入我的 hdfs 中。 我已经为开始我的工作制作了一个 Java 代码:
OozieClient wc = new OozieClient("http://localhost:11000/oozie");
Properties conf = wc.createConfiguration();
conf.setProperty( OozieClient.APP_PATH, "hdfs://hdfs_server:54310/hive/testSqoop/sqoop-workflow.xml" );
conf.setProperty("queueName", "default");
try {
String jobId = wc.run(conf);
System.out.println("Workflow job submitted");
while (wc.getJobInfo(jobId).getStatus() == WorkflowJob.Status.RUNNING) {
System.out.println("Workflow job running ...");
System.out.println("..." + wc.getJobInfo(jobId).getStatus().toString() );
Thread.sleep(10 * 1000);
}
System.out.println("Workflow job completed ...");
System.out.println(wc.getJobInfo(jobId));
} catch (Exception r) {
r.printStackTrace();
}
在 Oozie 网络界面中,我可以看到我的工作正在运行
2013-05-28 12:42:30,004 INFO ActionStartXCommand:539 - USER[anthonyc] GROUP[-] TOKEN[] APP[Sqoop] JOB[0000000-130528124140043-oozie-anth-W] ACTION[0000000-130528124140043-oozie-anth-W@:start:] Start action [0000000-130528124140043-oozie-anth-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2013-05-28 12:42:30,008 WARN ActionStartXCommand:542 - USER[anthonyc] GROUP[-] TOKEN[] APP[Sqoop] JOB[0000000-130528124140043-oozie-anth-W] ACTION[0000000-130528124140043-oozie-anth-W@:start:] [***0000000-130528124140043-oozie-anth-W@:start:***]Action status=DONE
2013-05-28 12:42:30,009 WARN ActionStartXCommand:542 - USER[anthonyc] GROUP[-] TOKEN[] APP[Sqoop] JOB[0000000-130528124140043-oozie-anth-W] ACTION[0000000-130528124140043-oozie-anth-W@:start:] [***0000000-130528124140043-oozie-anth-W@:start:***]Action updated in DB!
2013-05-28 12:42:30,192 INFO ActionStartXCommand:539 - USER[anthonyc] GROUP[-] TOKEN[] APP[Sqoop] JOB[0000000-130528124140043-oozie-anth-W] ACTION[0000000-130528124140043-oozie-anth-W@Sqoop] Start action [0000000-130528124140043-oozie-anth-W@Sqoop] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2013-05-28 12:42:31,389 WARN SqoopActionExecutor:542 - USER[anthonyc] GROUP[-] TOKEN[] APP[Sqoop] JOB[0000000-130528124140043-oozie-anth-W] ACTION[0000000-130528124140043-oozie-anth-W@Sqoop] credentials is null for the action
2013-05-28 12:42:42,942 INFO SqoopActionExecutor:539 - USER[anthonyc] GROUP[-] TOKEN[] APP[Sqoop] JOB[0000000-130528124140043-oozie-anth-W] ACTION[0000000-130528124140043-oozie-anth-W@Sqoop] checking action, external ID [job_1369126414383_0003] status [RUNNING]
2013-05-28 12:42:42,945 WARN ActionStartXCommand:542 - USER[anthonyc] GROUP[-] TOKEN[] APP[Sqoop] JOB[0000000-130528124140043-oozie-anth-W] ACTION[0000000-130528124140043-oozie-anth-W@Sqoop] [***0000000-130528124140043-oozie-anth-W@Sqoop***]Action status=RUNNING
2013-05-28 12:42:42,946 WARN ActionStartXCommand:542 - USER[anthonyc] GROUP[-] TOKEN[] APP[Sqoop] JOB[0000000-130528124140043-oozie-anth-W] ACTION[0000000-130528124140043-oozie-anth-W@Sqoop] [***0000000-130528124140043-oozie-anth-W@Sqoop***]Action updated in DB!
2013-05-28 12:47:43,034 INFO KillXCommand:539 - USER[anthonyc] GROUP[-] TOKEN[] APP[Sqoop] JOB[0000000-130528124140043-oozie-anth-W] ACTION[-] STARTED WorkflowKillXCommand for jobId=0000000-130528124140043-oozie-anth-W
2013-05-28 12:47:43,328 WARN CoordActionUpdateXCommand:542 - USER[anthonyc] GROUP[-] TOKEN[] APP[Sqoop] JOB[0000000-130528124140043-oozie-anth-W] ACTION[-] E1100: Command precondition does not hold before execution, [, coord action is null], Error Code: E1100
2013-05-28 12:47:43,328 INFO KillXCommand:539 - USER[anthonyc] GROUP[-] TOKEN[] APP[Sqoop] JOB[0000000-130528124140043-oozie-anth-W] ACTION[-] ENDED WorkflowKillXCommand for jobId=0000000-130528124140043-oozie-anth-W
当我检查 yarn 网络界面时,我可以看到我的工作,但状态为 FAILED with
Application application_1369126414383_0003 failed 1 times due to AM Container for appattempt_1369126414383_0003_000001 exited with exitCode: 1 due to: .Failing this attempt.. Failing the application.
我真的不知道哪里出了问题。 我需要你的建议。
谢谢~
最佳答案
您必须检查作业日志:
$ oozie job -log <coord_job_id>
了解正在发生的事情。
关于hadoop - Oozie 和 sqoop 配置问题,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/16804757/