apache - 带有Nutch的Apache Hadoop 2.7.4尝试由于任何原因失败

Question

我已经使用Nutch 2.3.1配置了Apache Hadoop 2.7.4，Hbase 1.2.6和Solr 6.6.1。我在所有节点上的mapred-site.xml是

<property>
  <name>mapreduce.jobtracker.address</name>
  <value>master:9001</value>
 </property>

 <property>
   <name>mapreduce.framework.name</name>
   <value>yarn</value>
 </property>

<property>
  <name>mapreduce.task.timeout</name>
  <value>10000</value>
  <description>The number of milliseconds before a task will be
  terminated if it neither reads an input, writes an output, nor
  updates its status string.  A value of 0 disables the timeout.
  </description>
</property>

<property>
  <name>mapreduce.reduce.memory.mb</name>
  <value>2048</value>
  <description>The amount of memory to request from the scheduler for each
  reduce task.
  </description>
</property>

和yarn-site.xml在所有节点上具有以下配置。

 <property>
      <name>yarn.resourcemanager.hostname</name>
      <value>master</value>
   </property>

  <property>
      <name>yarn.nodemanager.aux-services</name>
      <value>mapreduce_shuffle</value>
   </property>

  <property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>3072</value>
  </property>

  <property>
    <name>yarn.nodemanager.resource.cpu-vcores</name>
    <value>2</value>
  </property>

<property>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>1024</value>
  </property>

  <property>
    <name>yarn.scheduler.minimum-allocation-vcores</name>
    <value>1</value>
  </property>

所有作业均成功运行，但在solrindex时失败，并出现以下异常

17/10/13 12:30:43 INFO mapreduce.Job:  map 0% reduce 0%
17/10/13 12:35:12 INFO mapreduce.Job: Task Id : attempt_1507879413631_0001_m_000000_1, Status : FAILED
AttemptID:attempt_1507879413631_0001_m_000000_1 Timed out after 10 secs
17/10/13 12:39:42 INFO mapreduce.Job: Task Id : attempt_1507879413631_0001_m_000000_2, Status : FAILED
AttemptID:attempt_1507879413631_0001_m_000000_2 Timed out after 10 secs
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

17/10/13 12:44:13 INFO mapreduce.Job:  map 100% reduce 0%
17/10/13 12:44:13 INFO mapreduce.Job: Job job_1507879413631_0001 failed with state FAILED due to: Task failed task_1507879413631_0001_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

从纱日志之一具有以下信息

2017-10-13 12:30:12,442 INFO [IPC Server handler 0 on 37429] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1507879413631_0001_m_000000_0 is : 0.0
2017-10-13 12:30:15,450 INFO [IPC Server handler 5 on 37429] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1507879413631_0001_m_000000_0 is : 0.0
2017-10-13 12:30:41,870 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1507879413631_0001_m_000000_0: AttemptID:attempt_1507879413631_0001_m_000000_0 Timed out after 10 secs
2017-10-13 12:30:41,871 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1507879413631_0001_m_000000_0 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP
2017-10-13 12:30:41,872 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1507879413631_0001_01_000002 taskAttempt attempt_1507879413631_0001_m_000000_0
2017-10-13 12:30:41,873 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1507879413631_0001_m_000000_0
2017-10-13 12:30:41,873 INFO [ContainerLauncher #1] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : node1:39015
2017-10-13 12:30:41,896 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1507879413631_0001_m_000000_0 TaskAttempt Transitioned from FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
2017-10-13 12:30:41,896 INFO [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
2017-10-13 12:30:41,897 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1507879413631_0001_m_000000_0 TaskAttempt Transitioned from FAIL_TASK_CLEANUP to FAILED
2017-10-13 12:30:41,902 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved node3 to /default-rack
2017-10-13 12:30:41,903 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node node1
2017-10-13 12:30:41,904 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1507879413631_0001_m_000000_1 TaskAttempt Transitioned from NEW to UNASSIGNED
2017-10-13 12:30:41,905 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1507879413631_0001_m_000000_1 to list of failed maps
2017-10-13 12:30:42,847 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:1
2017-10-13 12:30:42,865 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1507879413631_0001: ask=1 release= 0 newContainers=0 finishedContainers=1 resourcelimit=<memory:7168, vCores:1> knownNMs=3
2017-10-13 12:30:42,865 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1507879413631_0001_01_000002
2017-10-13 12:30:42,866 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:1
2017-10-13 12:30:42,866 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1507879413631_0001_m_000000_0: Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

哪里有问题。我已经检查了所有日志，但是没有错误消息比上面的错误之一。其次，当我在独立计算机上运行此作业(伪模式)时，它已完成，没有任何问题。

Answer 1

您可以在日志中看到以下错误:
AttemptID:attempt_1507879413631_0001_m_000000_0 Timed out after 10 secs
您可以通过在此处修改设置来尝试增加超时:

<property>
  <name>mapreduce.task.timeout</name>
  <value>10000</value>
  <description>The number of milliseconds before a task will be
  terminated if it neither reads an input, writes an output, nor
  updates its status string.  A value of 0 disables the timeout.
  </description>
</property>

您的时间设置为10秒，通常使用600的超时时间(10分钟)。因此您可以将其更改为600000。