我正在尝试通过 Sqoop 将包含 3200 万条记录的表从 SQL Server 导入 Hive。连接是 SQL Server 成功的。但是 Map/Reduce 作业没有成功执行。它给出了以下错误:
18/07/19 04:00:11 INFO client.RMProxy: Connecting to ResourceManager at /127.0.0.1:8032
18/07/19 04:00:27 DEBUG db.DBConfiguration: Fetching password from job credentials store
18/07/19 04:00:27 INFO db.DBInputFormat: Using read commited transaction isolation
18/07/19 04:00:27 DEBUG db.DataDrivenDBInputFormat: Creating input split with lower bound '1=1' and upper bound '1=1'
18/07/19 04:00:28 INFO mapreduce.JobSubmitter: number of splits:1
18/07/19 04:00:29 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1531917395459_0002
18/07/19 04:00:30 INFO impl.YarnClientImpl: Submitted application application_1531917395459_0002
18/07/19 04:00:30 INFO mapreduce.Job: The url to track the job: http://quickstart.cloudera:8088/proxy/application_1531917395459_0002/
18/07/19 04:00:30 INFO mapreduce.Job: Running job: job_1531917395459_0002
18/07/19 04:43:02 INFO mapreduce.Job: Job job_1531917395459_0002 running in uber mode : false
18/07/19 04:43:03 INFO mapreduce.Job: map 0% reduce 0%
18/07/19 04:43:04 INFO mapreduce.Job: Job job_1531917395459_0002 failed with state FAILED due to: Application application_1531917395459_0002 failed 2 times due to ApplicationMaster for attempt appattempt_1531917395459_0002_000002 timed out. Failing the application.
18/07/19 04:43:08 INFO mapreduce.Job: Counters: 0
18/07/19 04:43:08 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
18/07/19 04:43:09 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 2,576.6368 seconds (0 bytes/sec)
18/07/19 04:43:10 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
18/07/19 04:43:10 INFO mapreduce.ImportJobBase: Retrieved 0 records.
18/07/19 04:43:10 ERROR tool.ImportTool: Error during import: Import job failed!
这里是来自 yarn-site.xml 文件的配置
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.dispatcher.exit-on-error</name>
<value>true</value>
</property>
<property>
<description>List of directories to store localized files in.</description>
<name>yarn.nodemanager.local-dirs</name>
<value>/var/lib/hadoop-yarn/cache/${user.name}/nm-local-dir</value>
</property>
<property>
<description>Where to store container logs.</description>
<name>yarn.nodemanager.log-dirs</name>
<value>/var/log/hadoop-yarn/containers</value>
</property>
<property>
<description>Where to aggregate logs to.</description>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/var/log/hadoop-yarn/apps</value>
</property>
<property>
<description>Classpath for typical applications.</description>
<name>yarn.application.classpath</name>
<value>
$HADOOP_CONF_DIR,
$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,
$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*
</value>
</property>
<!-- added by me -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>localhost</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>127.0.0.1:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>127.0.0.1:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>127.0.0.1:8031</value>
</property>
</configuration>
起初，作业在通过 0.0.0.0:8032 与资源管理器连接时卡住了。于是我把主机改成了127.0.0.1。然后行刑继续进行。但随后出现上述错误。即使我尝试只用 1000 行来执行这项工作，但同样的错误。此外，有时工作会被终止。
这是我的 sqoop 命令
sqoop import --connect "jdbc:sqlserver://system-ip;databaseName=TEST" --driver com.microsoft.sqlserver.jdbc.SQLServerDriver --username user1 --password password --hive-import --create-hive-table --hive-table "customer_data_1000" --table "customer_data_1000" --split-by Account_Branch_Converted -m 1 --verbose
这是我的 docker 命令以防万一:
docker run --hostname=quickstart.cloudera --privileged=true -t -p 127.0.0.1:8888:8888 -p 127.0.
0.1:7180:7180 -p 127.0.0.1:50070:50070 -i 7c41929668d8/usr/bin/docker-quickstart
这是资源管理器日志:
2018-07-26 07:18:26,439 INFO org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: Expired:appattempt_1532588462827_0001_000001 Timed out after 600 secs
2018-07-26 07:24:03,059 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Updating application attempt appattempt_1532588462827_0001_000001 with final state: FAILED, and exit status: -1000
2018-07-26 07:35:46,609 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1532588462827_0001_000001 State change from LAUNCHED to FINAL_SAVING
2018-07-26 07:35:49,502 INFO org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: Expired:quickstart.cloudera:36003 Timed out after 600 secs
2018-07-26 07:39:44,485 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Deactivating Node quickstart.cloudera:36003 as it is now LOST
2018-07-26 07:44:39,238 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: quickstart.cloudera:36003 Node Transitioned from RUNNING to LOST
2018-07-26 07:45:09,895 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Unregistering app attempt : appattempt_1532588462827_0001_000001
2018-07-26 07:49:43,848 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: Node not found resyncing quickstart.cloudera:36003
2018-07-26 07:49:43,916 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Application finished, removing password for appattempt_1532588462827_0001_000001
2018-07-26 07:49:45,738 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1532588462827_0001_000001 State change from FINAL_SAVING to FAILED
2018-07-26 07:49:47,095 WARN org.apache.hadoop.ipc.Server: IPC Server handler 12 on 8032, call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplicationReport from 127.0.0.1:45162 Call#608 Retry#0: output error
2018-07-26 07:49:47,100 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: The number of failed attempts is 1. The max attempts is 2
2018-07-26 07:49:47,887 INFO org.apache.hadoop.ipc.Server: IPC Server handler 12 on 8032 caught an exception
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:265)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:474)
at org.apache.hadoop.ipc.Server.channelWrite(Server.java:2621)
at org.apache.hadoop.ipc.Server.access$1900(Server.java:134)
at org.apache.hadoop.ipc.Server$Responder.processResponse(Server.java:989)
at org.apache.hadoop.ipc.Server$Responder.doRespond(Server.java:1054)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2141)
2018-07-26 07:49:49,127 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Registering app attempt : appattempt_1532588462827_0001_000002
2018-07-26 07:49:49,127 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1532588462827_0001_000002 State change from NEW to SUBMITTED
2018-07-26 07:49:49,127 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Cleaning master appattempt_1532588462827_0001_000001
2018-07-26 07:49:50,458 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1532588462827_0001_01_000001 Container Transitioned from RUNNING to KILLED
2018-07-26 07:49:50,459 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt: Completed container: container_1532588462827_0001_01_000001 in state: KILLED event:KILL
2018-07-26 07:49:50,460 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1532588462827_0001 CONTAINERID=container_1532588462827_0001_01_000001
2018-07-26 07:49:50,550 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Released container container_1532588462827_0001_01_000001 of capacity <memory:2048, vCores:1> on host quickstart.cloudera:36003, which currently has 0 containers, <memory:0, vCores:0> used and <memory:8192, vCores:8> available, release resources=true
2018-07-26 07:49:50,563 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Application attempt appattempt_1532588462827_0001_000001 released container container_1532588462827_0001_01_000001 on node: host: quickstart.cloudera:36003 #containers=0 available=8192 used=0 with event: KILL
2018-07-26 07:49:50,580 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Removed node quickstart.cloudera:36003 cluster capacity: <memory:0, vCores:0>
2018-07-26 07:49:50,580 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Application appattempt_1532588462827_0001_000001 is done. finalState=FAILED
2018-07-26 07:49:50,581 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo: Application application_1532588462827_0001 requests cleared
2018-07-26 07:49:51,860 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Added Application Attempt appattempt_1532588462827_0001_000002 to scheduler from user: root
2018-07-26 07:49:52,125 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1532588462827_0001_000002 State change from SUBMITTED to SCHEDULED
2018-07-26 07:50:04,533 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: quickstart.cloudera:36003 Node Transitioned from NEW to RUNNING
2018-07-26 07:50:04,534 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Added node quickstart.cloudera:36003 cluster capacity: <memory:8192, vCores:8>
2018-07-26 07:50:05,883 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: received container statuses on node manager register :[container_id { app_attempt_id { application_id { id: 1 cluster_timestamp: 1532588462827 } attemptId: 1 } id: 1 } container_state: C_COMPLETE resource { memory: 2048 virtual_cores: 1 } priority { priority: 0 } diagnostics: "Container Killed by ResourceManager\nContainer killed on request. Exit code is 143\nContainer exited with a non-zero exit code 143\n" container_exit_status: -106 creation_time: 1532588806652]
2018-07-26 07:50:05,883 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: NodeManager from node quickstart.cloudera(cmPort: 36003 httpPort: 8042) registered with capability: <memory:8192, vCores:8>, assigned nodeId quickstart.cloudera:36003
2018-07-26 07:50:06,020 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Container container_1532588462827_0001_01_000001 completed with event FINISHED
2018-07-26 07:50:06,022 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1532588462827_0001_02_000001 Container Transitioned from NEW to ALLOCATED
2018-07-26 07:50:06,023 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root OPERATION=AM Allocated Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1532588462827_0001 CONTAINERID=container_1532588462827_0001_02_000001
2018-07-26 07:50:06,025 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Assigned container container_1532588462827_0001_02_000001 of capacity <memory:2048, vCores:1> on host quickstart.cloudera:36003, which has 1 containers, <memory:2048, vCores:1> used and <memory:6144, vCores:7> available after allocation
2018-07-26 07:50:06,025 INFO org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM: Sending NMToken for nodeId : quickstart.cloudera:36003 for container : container_1532588462827_0001_02_000001
2018-07-26 07:50:06,026 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1532588462827_0001_02_000001 Container Transitioned from ALLOCATED to ACQUIRED
2018-07-26 07:50:06,026 INFO org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM: Clear node set for appattempt_1532588462827_0001_000002
2018-07-26 07:50:06,026 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Storing attempt: AppId: application_1532588462827_0001 AttemptId: appattempt_1532588462827_0001_000002 MasterContainer: Container: [ContainerId: container_1532588462827_0001_02_000001, NodeId: quickstart.cloudera:36003, NodeHttpAddress: quickstart.cloudera:8042, Resource: <memory:2048, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 172.17.0.2:36003 }, ]
2018-07-26 07:50:06,026 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1532588462827_0001_000002 State change from SCHEDULED to ALLOCATED_SAVING
2018-07-26 07:50:06,026 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1532588462827_0001_000002 State change from ALLOCATED_SAVING to ALLOCATED
2018-07-26 07:50:06,026 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Launching masterappattempt_1532588462827_0001_000002
2018-07-26 07:50:06,027 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Setting up container Container: [ContainerId: container_1532588462827_0001_02_000001, NodeId: quickstart.cloudera:36003, NodeHttpAddress: quickstart.cloudera:8042, Resource: <memory:2048, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 172.17.0.2:36003 }, ] for AM appattempt_1532588462827_0001_000002
2018-07-26 07:50:06,027 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Command to launch container container_1532588462827_0001_02_000001 : $JAVA_HOME/bin/java -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Xmx1024m org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
2018-07-26 07:50:06,027 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Create AMRMToken for ApplicationAttempt: appattempt_1532588462827_0001_000002
2018-07-26 07:50:06,027 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Creating password for appattempt_1532588462827_0001_000002
2018-07-26 07:50:06,128 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Done launching container Container: [ContainerId: container_1532588462827_0001_02_000001, NodeId: quickstart.cloudera:36003, NodeHttpAddress: quickstart.cloudera:8042, Resource: <memory:2048, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 172.17.0.2:36003 }, ] for AM appattempt_1532588462827_0001_000002
2018-07-26 07:50:06,129 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1532588462827_0001_000002 State change from ALLOCATED to LAUNCHED
2018-07-26 07:50:06,953 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1532588462827_0001_02_000001 Container Transitioned from ACQUIRED to RUNNING
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Container container_1532588462827_0001_01_000001 completed with event FINISHED
2018-07-26 07:50:32,951 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1532588462827_0001_000002 (auth:SIMPLE)
2018-07-26 07:50:33,014 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: AM registration appattempt_1532588462827_0001_000002
emanager.scheduler.fair.FairScheduler: Container container_1532588462827_0001_01_000001 completed with event FINISHED
2018-07-26 07:50:34,887 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Container container_1532588462827_0001_01_000001 completed with event FINISHED
2018-07-26 07:50:34,893 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1532588462827_0001_02_000002 Container Transitioned from NEW to ALLOCATED
2018-07-26 07:50:34,893 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root OPERATION=AM Allocated Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1532588462827_0001 CONTAINERID=container_1532588462827_0001_02_000002
2018-07-26 07:50:34,894 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Assigned container container_1532588462827_0001_02_000002 of capacity <memory:1024, vCores:1> on host quickstart.cloudera:36003, which has 2 containers, <memory:3072, vCores:2> used and <memory:5120, vCores:6> available after allocation
2018-07-26 07:50:36,467 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Container container_1532588462827_0001_01_000001 completed with event FINISHED
2018-07-26 07:50:36,478 INFO org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM: Sending NMToken for nodeId : quickstart.cloudera:36003 for container : container_1532588462827_0001_02_000002
2018-07-26 07:50:36,479 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1532588462827_0001_02_000002 Container Transitioned from ALLOCATED to ACQUIRED
2018-07-26 07:50:36,898 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Container container_1532588462827_0001_01_000001 completed with event FINISHED
2018-07-26 07:50:38,113 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1532588462827_0001_02_000002 Container Transitioned from ACQUIRED to RUNNING
2018-07-26 07:50:38,113 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo: checking for deactivate...
2018-07-26 07:50:54,379 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1532588462827_0001_02_000002 Container Transitioned from RUNNING to COMPLETED
2018-07-26 07:50:54,525 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt: Completed container: container_1532588462827_0001_02_000002 in state: COMPLETED event:FINISHED
2018-07-26 07:50:54,553 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1532588462827_0001 CONTAINERID=container_1532588462827_0001_02_000002
2018-07-26 07:50:54,555 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Released container container_1532588462827_0001_02_000002 of capacity <memory:1024, vCores:1> on host quickstart.cloudera:36003, which currently has 1 containers, <memory:2048, vCores:1> used and <memory:6144, vCores:7> available, release resources=true
2018-07-26 07:50:54,558 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Application attempt appattempt_1532588462827_0001_000002 released container container_1532588462827_0001_02_000002 on node: host: quickstart.cloudera:36003 #containers=1 available=6144 used=2048 with event: FINISHED
2018-07-26 07:50:55,386 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Container container_1532588462827_0001_02_000002 completed with event FINISHED
438 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt: Completed container: container_1532588462827_0001_02_000001 in state: COMPLETED event:FINISHED
2018-07-26 07:51:00,438 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1532588462827_0001 CONTAINERID=container_1532588462827_0001_02_000001
2018-07-26 07:51:00,438 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Released container container_1532588462827_0001_02_000001 of capacity <memory:2048, vCores:1> on host quickstart.cloudera:36003, which currently has 0 containers, <memory:0, vCores:0> used and <memory:8192, vCores:8> available, release resources=true
2018-07-26 07:51:00,438 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Updating application attempt appattempt_1532588462827_0001_000002 with final state: FAILED, and exit status: 0
2018-07-26 07:51:00,439 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1532588462827_0001_000002 State change from RUNNING to FINAL_SAVING
2018-07-26 07:51:00,439 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Unregistering app attempt : appattempt_1532588462827_0001_000002
2018-07-26 07:51:00,439 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Application finished, removing password for appattempt_1532588462827_0001_000002
2018-07-26 07:51:00,439 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1532588462827_0001_000002 State change from FINAL_SAVING to FAILED
2018-07-26 07:51:00,439 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: The number of failed attempts is 2. The max attempts is 2
2018-07-26 07:51:00,439 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Updating application application_1532588462827_0001 with final state: FAILED
2018-07-26 07:51:00,457 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1532588462827_0001 State change from RUNNING to FINAL_SAVING
2018-07-26 07:51:00,458 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Updating info for app: application_1532588462827_0001
2018-07-26 07:51:00,458 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Application appattempt_1532588462827_0001_000002 is done. finalState=FAILED
2018-07-26 07:51:00,458 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo: Application application_1532588462827_0001 requests cleared
2018-07-26 07:51:00,458 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Cleaning master appattempt_1532588462827_0001_000002
2018-07-26 07:51:05,760 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Application application_1532588462827_0001 failed 2 times due to AM Container for appattempt_1532588462827_0001_000002 exited with exitCode: 0
For more detailed output, check application tracking page:http://quickstart.cloudera:8088/proxy/application_1532588462827_0001/Then, click on links to logs of each attempt.
Diagnostics: Failing this attempt. Failing the application.
2018-07-26 07:51:05,781 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1532588462827_0001 State change from FINAL_SAVING to FAILED
2018-07-26 07:51:05,785 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App failed with state: FAILED PERMISSIONS=Application application_1532588462827_0001 failed 2 times due to AM Container for appattempt_1532588462827_0001_000002 exited with exitCode: 0
For more detailed output, check application tracking page:http://quickstart.cloudera:8088/proxy/application_1532588462827_0001/Then, click on links to logs of each attempt.
Diagnostics: Failing this attempt. Failing the application. APPID=application_1532588462827_0001
2018-07-26 07:51:05,819 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary: appId=application_1532588462827_0001,name=customer_data_1000.jar,user=root,queue=root.root,state=FAILED,trackingUrl=http://quickstart.cloudera:8088/cluster/app/application_1532588462827_0001,appMasterHost=N/A,startTime=1532588804719,finishTime=1532591460451,finalStatus=FAILED
2018-07-26 07:51:05,821 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Container container_1532588462827_0001_02_000001 completed with event FINISHED
2018-07-26 07:51:05,822 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Container container_1532588462827_0001_02_000002 completed with event FINISHED
我哪里错了？
最佳答案
我不能给你一个精确的解决方案，但我能做的就是告诉你根本原因是什么:
- 尝试以非 root 用户身份运行 Sqoop 作业。
- 检查您的 JDK 是否正确安装在主机上，您的 JAVA_HOME 是否设置正确。
- 检查您是否已授予您正在使用的数据库的正确权限。
由于上述原因之一，您的工作失败了。由于您有足够的 v-cores 和可用内存，并且容器也正在创建中。所以，处理端一切正常，但肯定有配置错误。
关于docker - Sqoop - 导入作业失败，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/51414922/