java - 如何将文件作为输入并在多线程中工作？

Question

我有这段代码来了解如何从 URL 获取状态代码:

import java.io.IOException;
import java.net.HttpURLConnection;
import java.net.URL;

/**
 * @author Crunchify.com
 * 
 */

 class j {
    public static void main(String args[]) throws Exception {

        String[] hostList = { "http://example.com", "http://example2.com","http://example3.com" };

        for (int i = 0; i < hostList.length; i++) {

            String url = hostList[i];
            String status = getStatus(url);

            System.out.println(url + "\t\tStatus:" + status);
        }
    }

    public static String getStatus(String url) throws IOException {

        String result = "";
        try {
            URL siteURL = new URL(url);
            HttpURLConnection connection = (HttpURLConnection) siteURL
                    .openConnection();
            connection.setRequestMethod("HEAD");
            connection.connect();

            int code = connection.getResponseCode();

                result = Integer.toString(code);

        } catch (Exception e) {
            result = "->Red<-";
        }
        return result;
    }
}

我已经检查了它的小输入，它工作正常。但是我有数百万个域需要扫描。我有一个包含它的文件。

1. 我想知道如何将文件作为此代码的输入。
2. 我希望代码在多线程中工作。说 Thread count 应该超过 20000，这样我的输出会更快。
3. 如何将输出写入另一个文件？

请帮助我。如果可能的话，我想知道哪种 Bandwidth Savvy 方法可以完成同样的工作。无论如何，我想让代码更快。我如何用我的代码做这些事情？ Java 版本:

java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

Answer 1

这就是你想要的:

输入列表文件(c://lines.txt)

http://www.adam-bien.com/
http://stackoverflow.com/
http://www.dfgdfgdfgdfgdfgertwsgdfhdfhsru.de
http://www.google.de

主题:

import java.net.HttpURLConnection;
import java.net.URL;
import java.util.concurrent.Callable;

public class StatusThread implements Callable<String> {

    String url;

    public StatusThread(String url) {
        this.url = url;
    }

    @Override
    public String call() throws Exception {

        String result = "";
        try {
            URL siteURL = new URL(url);
            HttpURLConnection connection = (HttpURLConnection) siteURL.openConnection();
            connection.setRequestMethod("HEAD");
            connection.connect();

            int code = connection.getResponseCode();

            result = Integer.toString(code);

        } catch (Exception e) {
            result = "->Red<-";
        }
        return url + "|" + result;
    }
}

和主程序:

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.stream.Stream;

public class CallableExample {
    public static void main(String[] args) throws IOException {

        // Number of threads
        int numberOfThreads = 10;

        // Input file
        String sourceFileName = "c://lines.txt"; // Replace by your own
        String targetFileName = "c://output.txt"; // Replace by your own

        // Read input file into List    
        ArrayList<String> urls = new ArrayList<>();
        try (Stream<String> stream = Files.lines(Paths.get(sourceFileName ))) {
            stream.forEach((string) -> {
                urls.add(string);
            });

        } catch (IOException e) {
            e.printStackTrace();
        }

        // Create thread pool
        ThreadPoolExecutor executor = (ThreadPoolExecutor) Executors.newFixedThreadPool(numberOfThreads);
        List<Future<String>> resultList = new ArrayList<>();

        // Launch threads
        for(String url : urls) {
            StatusThread statusGetter = new StatusThread(url);
            Future<String> result = executor.submit(statusGetter);
            resultList.add(result);
        }

        // Use results
        FileWriter writer;
        writer = new FileWriter(targetFileName);
        for (Future<String> future : resultList) {
            try {
                String oneResult = future.get().split("\\|")[0] + " -> " + future.get().split("\\|")[1];

                // Print the results to the console
                System.out.println(oneResult);

                // Write the result to a file
                writer.write(oneResult + System.lineSeparator());

            } catch (InterruptedException | ExecutionException e) {
                e.printStackTrace();
            }
        }
        writer.close();


        // Shut down the executor service
        executor.shutdown();
    }
}

不要忘记:

• 创建您的输入文件并指向它 (c://lines.txt)
• 更改线程数以获得最佳结果