我正在通过包括 sqoop 1.4.6 的 Cloudera 5.8.0 使用 Hive/Hadoop/Sqoop。我的 Hadoop 集群有 4 个 Hadoop 数据节点，每个节点有 16 GB 内存，并且都在运行 ImpalaDaemons 和 Yarn NodeManagers。 Yarn 服务器与 Hue、Hive 和 Sqoop2 一起在具有 32 GB RAM(具有多种角色)的服务器上运行。
使用 Sqoop 从 MySQL 数据库导入(从使用 Sqoop 1 的主服务器通过 bash 脚本导入到增量作业中的 parquetfile 格式)，即使导入只有 200 行(甚至 30 行)的表，它似乎也很慢(平均 50 秒)在一种情况下)。即使在 Ubermode 中，它也会在 Sqoop 的这一步上始终挂起(并最终成功)持续 30 秒:
注:清洁阶段和重复省略为简洁..
2016-11-03 10:07:50,534 WARN org.apache.hadoop.hdfs.BlockReaderFactory: I/O error constructing remote block reader.
java.io.IOException: Got error for OP_READ_BLOCK, status=ERROR, self=/192.168.1.31:58178, remote=/192.168.1.34:50010, for file /user/(user profile name)/.staging/job_1478124814973_0001/libjars/commons-math-2.1.jar, for pool BP-15528599-192.168.1.31-1472851278753 block 1074078887_338652
at org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockReader2.java:467)
at org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:432)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:881)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:759)
at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:376)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:662)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:889)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:942)
at java.io.DataInputStream.read(DataInputStream.java:100)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:369)
at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:265)
at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:356)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-11-03 10:07:50,541 WARN org.apache.hadoop.hdfs.DFSClient: Failed to connect to /192.168.1.34:50010 for block, add to deadNodes and continue. java.io.IOException: Got error for OP_READ_BLOCK, status=ERROR, self=/192.168.1.31:58178, remote=/192.168.1.34:50010, for file /user/(user profile name)/.staging/job_1478124814973_0001/libjars/commons-math-2.1.jar, for pool BP-15528599-192.168.1.31-1472851278753 block 1074078887_338652
java.io.IOException: Got error for OP_READ_BLOCK, status=ERROR, self=/192.168.1.31:58178, remote=/192.168.1.34:50010, for file /user/(user profile name)/.staging/job_1478124814973_0001/libjars/commons-math-2.1.jar, for pool BP-15528599-192.168.1.31-1472851278753 block 1074078887_338652
at org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockReader2.java:467)
at org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:432)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:881)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:759)
at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:376)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:662)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:889)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:942)
at java.io.DataInputStream.read(DataInputStream.java:100)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:369)
at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:265)
at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:356)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-11-03 10:07:50,543 INFO org.apache.hadoop.hdfs.DFSClient: Successfully connected to /192.168.1.33:50010 for BP-15528599-192.168.1.31-1472851278753:blk_1074078887_338652
这个错误重复了 4 次。
当再次运行这项工作时，我得到了这个:
2016-11-03 10:37:38,093 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e86_1478124814973_0002_01_000001 by user (user profile name)
2016-11-03 10:37:38,093 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1478124814973_0002
2016-11-03 10:37:38,095 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=(user profile name) IP=192.168.1.34 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478124814973_0002 CONTAINERID=container_e86_1478124814973_0002_01_000001
2016-11-03 10:37:38,096 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1478124814973_0002 transitioned from NEW to INITING
2016-11-03 10:37:38,096 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Adding container_e86_1478124814973_0002_01_000001 to application application_1478124814973_0002
2016-11-03 10:37:38,106 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: rollingMonitorInterval is set as -1. The log rolling mornitoring interval is disabled. The logs will be aggregated after this application is finished.
2016-11-03 10:37:38,134 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1478124814973_0002 transitioned from INITING to RUNNING
2016-11-03 10:37:38,138 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_e86_1478124814973_0002_01_000001 transitioned from NEW to LOCALIZING
2016-11-03 10:37:38,138 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1478124814973_0002
2016-11-03 10:37:38,147 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e86_1478124814973_0002_01_000001
2016-11-03 10:37:38,148 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /yarn/nm/nmPrivate/container_e86_1478124814973_0002_01_000001.tokens. Credentials list:
2016-11-03 10:37:38,149 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Initializing user (user profile name)
2016-11-03 10:37:38,151 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /yarn/nm/nmPrivate/container_e86_1478124814973_0002_01_000001.tokens to /yarn/nm/usercache/(user profile name)/appcache/application_1478124814973_0002/container_e86_1478124814973_0002_01_000001.tokens
2016-11-03 10:37:38,151 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Localizer CWD set to /yarn/nm/usercache/(user profile name)/appcache/application_1478124814973_0002 = file:/yarn/nm/usercache/(user profile name)/appcache/application_1478124814973_0002
2016-11-03 10:37:41,791 WARN org.apache.hadoop.hdfs.BlockReaderFactory: I/O error constructing remote block reader.
java.io.IOException: Got error for OP_READ_BLOCK, status=ERROR, self=/192.168.1.31:39276, remote=/192.168.1.35:50010, for file /user/(user profile name)/.staging/job_1478124814973_0002/libjars/jackson-core-2.3.1.jar, for pool BP-15528599-192.168.1.31-1472851278753 block 1074079133_338898
at org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockReader2.java:467)
at org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:432)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:881)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:759)
at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:376)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:662)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:889)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:942)
at java.io.DataInputStream.read(DataInputStream.java:100)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:369)
at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:265)
at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:356)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-11-03 10:37:41,792 WARN org.apache.hadoop.hdfs.DFSClient: Failed to connect to /192.168.1.35:50010 for block, add to deadNodes and continue. java.io.IOException: Got error for OP_READ_BLOCK, status=ERROR, self=/192.168.1.31:39276, remote=/192.168.1.35:50010, for file /user/(user profile name)/.staging/job_1478124814973_0002/libjars/jackson-core-2.3.1.jar, for pool BP-15528599-192.168.1.31-1472851278753 block 1074079133_338898
java.io.IOException: Got error for OP_READ_BLOCK, status=ERROR, self=/192.168.1.31:39276, remote=/192.168.1.35:50010, for file /user/(user profile name)/.staging/job_1478124814973_0002/libjars/jackson-core-2.3.1.jar, for pool BP-15528599-192.168.1.31-1472851278753 block 1074079133_338898
at org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockReader2.java:467)
at org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:432)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:881)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:759)
at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:376)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:662)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:889)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:942)
at java.io.DataInputStream.read(DataInputStream.java:100)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:369)
at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:265)
at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:61)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:357)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:356)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:60)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-11-03 10:37:41,795 INFO org.apache.hadoop.hdfs.DFSClient: Successfully connected to /192.168.1.32:50010 for BP-15528599-192.168.1.31-1472851278753:blk_1074079133_338898
2016-11-03 10:37:42,928 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_e86_1478124814973_0002_01_000001 transitioned from LOCALIZING to LOCALIZED
2016-11-03 10:37:42,951 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_e86_1478124814973_0002_01_000001 transitioned from LOCALIZED to RUNNING
2016-11-03 10:37:42,955 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, /yarn/nm/usercache/(user profile name)/appcache/application_1478124814973_0002/container_e86_1478124814973_0002_01_000001/default_container_executor.sh]
2016-11-03 10:37:43,011 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e86_1478124814973_0002_01_000001
2016-11-03 10:37:43,034 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25215 for container-id container_e86_1478124814973_0002_01_000001: 1.4 MB of 2 GB physical memory used; 103.6 MB of 4.2 GB virtual memory used
2016-11-03 10:37:46,242 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25215 for container-id container_e86_1478124814973_0002_01_000001: 268.1 MB of 2 GB physical memory used; 1.4 GB of 4.2 GB virtual memory used
2016-11-03 10:37:49,261 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25215 for container-id container_e86_1478124814973_0002_01_000001: 398.4 MB of 2 GB physical memory used; 1.5 GB of 4.2 GB virtual memory used
2016-11-03 10:37:52,279 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25215 for container-id container_e86_1478124814973_0002_01_000001: 408.5 MB of 2 GB physical memory used; 1.5 GB of 4.2 GB virtual memory used
2016-11-03 10:37:55,297 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25215 for container-id container_e86_1478124814973_0002_01_000001: 416.6 MB of 2 GB physical memory used; 1.5 GB of 4.2 GB virtual memory used
2016-11-03 10:37:58,315 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25215 for container-id container_e86_1478124814973_0002_01_000001: 414.1 MB of 2 GB physical memory used; 1.5 GB of 4.2 GB virtual memory used
2016-11-03 10:38:00,934 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container container_e86_1478124814973_0002_01_000001 succeeded
2016-11-03 10:38:00,934 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_e86_1478124814973_0002_01_000001 transitioned from RUNNING to EXITED_WITH_SUCCESS
2016-11-03 10:38:00,935 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e86_1478124814973_0002_01_000001
2016-11-03 10:38:00,967 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /yarn/nm/usercache/(user profile name)/appcache/application_1478124814973_0002/container_e86_1478124814973_0002_01_000001
2016-11-03 10:38:00,968 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=(user profile name) OPERATION=Container Finished - Succeeded TARGET=ContainerImpl RESULT=SUCCESS APPID=application_1478124814973_0002 CONTAINERID=container_e86_1478124814973_0002_01_000001
2016-11-03 10:38:00,968 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_e86_1478124814973_0002_01_000001 transitioned from EXITED_WITH_SUCCESS to DONE
2016-11-03 10:38:00,968 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Removing container_e86_1478124814973_0002_01_000001 from application application_1478124814973_0002
2016-11-03 10:38:00,968 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e86_1478124814973_0002_01_000001 for log-aggregation
2016-11-03 10:38:00,968 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Stopping container with container Id: container_e86_1478124814973_0002_01_000001
2016-11-03 10:38:00,980 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=(user profile name) IP=192.168.1.34 OPERATION=Stop Container Request
TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1478124814973_0002 CONTAINERID=container_e86_1478124814973_0002_01_000001
2016-11-03 10:38:01,316 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e86_1478124814973_0002_01_000001
2016-11-03 10:38:01,972 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e86_1478124814973_0002_01_000001]
2016-11-03 10:38:01,972 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1478124814973_0002 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP
2016-11-03 10:38:01,973 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /yarn/nm/usercache/(user profile name)/appcache/application_1478124814973_0002
2016-11-03 10:38:01,973 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1478124814973_0002
2016-11-03 10:38:01,973 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1478124814973_0002 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED
2016-11-03 10:38:01,973 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1478124814973_0002
2016-11-03 10:38:02,072 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e86_1478124814973_0002_01_000001. Current good log dirs are /yarn/container-logs
2016-11-03 10:38:02,073 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /yarn/container-logs/application_1478124814973_0002/container_e86_1478124814973_0002_01_000001/stderr
2016-11-03 10:38:02,074 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /yarn/container-logs/application_1478124814973_0002/container_e86_1478124814973_0002_01_000001/stdout
2016-11-03 10:38:02,074 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /yarn/container-logs/application_1478124814973_0002/container_e86_1478124814973_0002_01_000001/syslog
2016-11-03 10:38:02,160 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /yarn/container-logs/application_1478124814973_0002
经过后续测试，它在节点 3 和 4(192.168.1.34 和 192.168.1.35)上出现问题。 Cloudera 界面说所有节点都健康(我意识到它可能不准确)。我可以相信一个可能是坏的(并尝试将其分解并删除，然后重新编译它)但两个似乎很奇怪，尤其是当我可以毫无问题地查询 Impala 或 hive 并且 Cloudera 和 fsck 说节点是健康的。
我已经在根目录上运行 hdfs fsck 并且没有发现错误。任何人都明白为什么会发生这种情况，更好的是，这可以解决吗？
哦，应该注意所有节点都是同一物理服务器上的虚拟机，并且所有/etc/hosts 文件都配置为查看所有节点主机名(目前不使用内部 DNS)。我已经检查了 192.168.1.34 和 192.168.1.35 上的 iptables 服务，但 ip 表没有运行。还验证了两台机器都在监听端口 50010。
谢谢大家!
最佳答案
好的，设法摆脱了错误。我改变了这个设置:
mapreduce.client.submit.file.replication
将此值设置为 4(集群中的 yarn 节点管理器数量，之前为 2)I/O 异常错误消失。至于速度，研究表明，如果使用 Parquet 格式，带有小文件的小表格的处理效率很低。所以我猜我是否有较小的表导入中间表被转换为 HBase 文件格式(或者可能所有这些表都被用作从原始 sqoop 导入到带有时间戳列的 native 表格式的中间表(sqoop 将它们转换为 longint如果使用 Parquet 格式)。如果我制作这些 Hbase，我不再需要讽刺地转换它们。
关于apache - 由于 YARN 的旋转 hadoop 连接问题导致 SQOOP 1 导入缓慢，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/40406571/