java - 公开访问的 URL 抛出 IOException

Question

我想访问链接http://www.nation.co.ke/business/seedsofgold/Egg-imports-from-Uganda-hatch-big-losses-for-farmers/-/2301238/2897930/-/dpeqesz/-/index.html

该链接可公开访问，甚至可以使用 curl 加载

但是在Java代码中它会抛出线程“main”中的异常java.io.IOException:服务器返回HTTP响应代码:403 URL:http://www.nation.co.ke/business/seedsofgold/乌干达鸡蛋进口孵化损失大/-/2301238/2897930/-/dpeqesz/-/index.html

这是代码:

/**
 * 
 * @param url the HTML page
 * @throws IOException
 */
public static String getPage(String url) throws IOException {
    URL u = new URL(url);
    URLConnection conn = u.openConnection();

    String mime = conn.getContentType();
    if( !StringUtils.containsIgnoreCase(mime, "text/html") ) {
        return null; // don't continue if not HTML
    }
    else {

        // read the response body, using BufferedReader for performance
        InputStream in = conn.getInputStream();
        BufferedReader reader = new BufferedReader(new InputStreamReader(in, Charset.defaultCharset()));
        int n = 0, totalRead = 0;
        char[] buf = new char[1024];
        StringBuilder content = new StringBuilder();

        // read until EOF or first 16384 characters
        while (totalRead < 16384 && (n = reader.read(buf, 0, buf.length)) != -1) {
            content.append(buf, 0, n);
            totalRead += n;
        }
        reader.close();

}

错误发生在:

       InputStream in = conn.getInputStream();

相同的代码可以与其他 URL 配合使用。

Answer 1

尝试添加

conn.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");

在 URLConnection conn = u.openConnection(); 之后立即连接到您的连接。许多网站在未设置正确的代理时会阻止网站访问。