java - 如何修复 src 文件系统问题上的资源更改

Question

我尝试在 MR 上使用 Hive 执行 SQL，但中途失败并出现以下错误:

Application application_1570514228864_0001 failed 2 times due to AM Container for appattempt_1570514228864_0001_000002 exited with exitCode: -1000
Failing this attempt.Diagnostics: [2019-10-08 13:57:49.272]Failed to download resource { { s3a://tpcds/tmp/hadoop-yarn/staging/root/.staging/job_1570514228864_0001/libjars, 1570514262820, FILE, null },pending,[(container_1570514228864_0001_02_000001)],1132444167207544,DOWNLOADING} java.io.IOException: Resource s3a://tpcds/tmp/hadoop-yarn/staging/root/.staging/job_1570514228864_0001/libjars changed on src filesystem (expected 1570514262820, was 1570514269265

从我的角度来看，错误日志中的关键消息是 libjars 在 src 文件系统上发生了更改(预期为 1570514262820，实际为 1570514269265。SO 上有多个关于此问题的线程，但尚未得到解答，例如 thread1 和 thread2 。

我从apache jira中发现了一些有值(value)的东西和redhat bugzilla 。我通过所有相关节点通过 NTP 同步时钟。但同样的问题仍然存在。

欢迎任何评论，谢谢。

Answer 1

我仍然不知道为什么资源文件的时间戳不一致，并且没有办法通过配置方式修复它，AFAIK。

但是，我设法找到了解决方法来跳过该问题。让我在这里为可能遇到同样问题的人总结一下。

通过检查错误日志并在Hadoop源代码中搜索，我们可以在hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java处跟踪问题。

只需删除异常抛出语句，

  private void verifyAndCopy(Path destination)
      throws IOException, YarnException {
    final Path sCopy;
    try {
      sCopy = resource.getResource().toPath();
    } catch (URISyntaxException e) {
      throw new IOException("Invalid resource", e);
    }
    FileSystem sourceFs = sCopy.getFileSystem(conf);
    FileStatus sStat = sourceFs.getFileStatus(sCopy);
    if (sStat.getModificationTime() != resource.getTimestamp()) {
            /**
      throw new IOException("Resource " + sCopy +
          " changed on src filesystem (expected " + resource.getTimestamp() +
          ", was " + sStat.getModificationTime());
          **/
            LOG.debug("[Gearon][Info] The timestamp is not consistent among resource files.\n" +
                            "Stop throwing exception . It doesn't affect other modules. ");
    }
    if (resource.getVisibility() == LocalResourceVisibility.PUBLIC) {
      if (!isPublic(sourceFs, sCopy, sStat, statCache)) {
        throw new IOException("Resource " + sCopy +
            " is not publicly accessible and as such cannot be part of the" +
            " public cache.");
      }
    }

    downloadAndUnpack(sCopy, destination);
  }

构建 hadoop-yarn-project 并将“hadoop-yarn-common-x.x.x.jar复制到$HADOOP_HOME/share/hadoop/yarn`。

将此线程留在这里，感谢您提供有关如何在不更改 hadoop 源的情况下修复它的进一步说明。