java - 指定的目标目录不存在

Question

我正在尝试使用Hue界面提供的Oozie仪表板创建工作流。尝试逐步进行，我的工作流程只有一个Java步骤。此java步骤的相关代码如下:

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class InputPathsCalculator {

    private static final Logger LOGGER = LoggerFactory.getLogger(InputPathsCalculator.class);

    public static void main(String[] args) throws IOException {
    System.out.println("sout-ing");
    LOGGER.info("putting something in the log");
    JobConf jobConf = new JobConf();
    jobConf.addResource(new Path("/etc/hadoop/conf/hdfs-site.xml"));

    Path outputPath = new Path(args[1]);
    List<Path> inputPaths = calculateInputPaths(args[0], jobConf);
    FileUtil.copy(fileSystem,
                         inputPaths.toArray(new Path[0]),
                         fileSystem,
                         outputPath,
                         false,
                         true,
                         jobConf);
    }
}

calculateInputPaths(...)是一种经过单独测试的方法，效果很好。我传递给该方法的参数是一个配置文件和一个值String的/usr/myUser/outputs/。

我在这里有两个问题:
1.我在任何日志中都看不到任何东西。不是我放入控制台的内容，不是我放入日志的内容
2. outputs目录存在，但是我得到以下堆栈跟踪:

org.apache.oozie.action.hadoop.JavaMainException: java.io.IOException: `/user/eliasg/outputs/output': specified destination directory does not exist
  at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:58)
  at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:39)
  at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:36)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:415)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.io.IOException: `/user/eliasg/outputs/output': specified destination directory does not exist
  at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:306)
  at com.ig.hadoop.jsonextractor.InputPathsCalculator.main(InputPathsCalculator.java:37)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:55)
  ... 15 more

我有一种感觉，对于第二点，我的jobConfig丢失了一些可以与hdfs一起使用的东西，但是我不知道是什么。关于第一点，我完全迷失了。

Answer 1

我在看错地方了。日志在那里，但是显然我需要查看 map task 容器日志而不是应用程序日志列表。

我发现仅将hdfs-site.xml添加到jobConf中是不够的。有一个默认情况下未启用的属性，需要设置。该属性是:jobConf.set("fs.default.name", String.format("hdfs://%1$s", jobConf.get("dfs.nameservices")));