java - 用于从 hdfs 提供输入并将输出写入 excel 文件的 Hadoop Mapreduce 示例

Question

<分区>

Answer 1

您只需使用 Java 和 Excel 执行必要的步骤，即可使用 Hadoop 正确处理您的信息。

• 上传或添加文件到 HDFS

这里有一个关于如何进行输入的典型示例:

    public void addFile(String source, String dest) throws IOException {     

    // Conf object will read the HDFS configuration parameters
    Configuration conf = new Configuration();
    conf.addResource(new Path("/home/hadoop/hadoop/conf/core-site.xml"));
    conf.addResource(new Path("/home/hadoop/hadoop/conf/hdfs-site.xml"));
    conf.addResource(new Path("/home/hadoop/hadoop/conf/mapred-site.xml"));

    FileSystem fileSystem = FileSystem.get(conf);

    // Get the filename out of the file path
    String filename = source.substring(source.lastIndexOf('/') + 1, source.length());

    // Create the destination path including the filename.
    if (dest.charAt(dest.length() - 1) != '/') {
    dest = dest + "/" + filename;
    } else {
    dest = dest + filename;
    }

    // Check if the file already exists
    Path path = new Path(dest);
    if (fileSystem.exists(path)) {
    System.out.println("File " + dest + " already exists");
    return;
    }

    // Create a new file and write data to it.
    FSDataOutputStream out = fileSystem.create(path);
    InputStream in = new BufferedInputStream(new FileInputStream(
    new File(source)));

    byte[] b = new byte[1024];
    int numBytes = 0;
    while ((numBytes = in.read(b)) > 0) {
    out.write(b, 0, numBytes);
    }

    // Close all the file descripters
    in.close();
    out.close();
    fileSystem.close();
}

来源:A HDFSClient for Hadoop - Linux Junkies

然后按照有关如何可视化数据输出的说明进行操作:

• Import the Server Log Data into Excel .

更多有用的信息:

• Running Hadoop on Ubuntu Linux

Hadoop 通过轻松集成为您提供所有可能需要的工具，以优化您的数据分析和操作。