java - 该网站不显示 chromedriver/geckodriver (Selenium) 中的信息

Question

我一直在尝试废弃 https://wizzair.com/en-gb/flights/timetable#/ 。 一段时间进展顺利。但从今天开始，我无法获取航类信息，因为 Wizz 服务器没有返回任何内容，并显示“发生错误。请重试。如果错误仍然存​​在，请联系航空公司。”错误。

我尝试不是从 Selenium 而是从 .exe 手动访问该网站，对于 geckodriver 和 chrome 来说都是一样的。 所以，看起来该网站知道它是一个自动管理的工具，并且不返回任何信息。

您对如何解决这个问题有什么建议吗？

谢谢

更新: 可以在此处找到使用或不使用 WebDriver 访问的已保存网页: https://drive.google.com/drive/folders/1OsqfKqKyqpOLBMdUbunYH7GUQZgqtRXJ?usp=sharing

代码试用

    System.setProperty("webdriver.gecko.driver", ".\\resources\\drivers\\geckodriver.exe");
    System.setProperty(FirefoxDriver.SystemProperty.DRIVER_USE_MARIONETTE, "true");
    System.setProperty(FirefoxDriver.SystemProperty.BROWSER_LOGFILE, "/dev/null");

    WebDriver driver = new FirefoxDriver();
    driver.manage().window().setPosition(new Point(2000, 0)); // move window to the second display
    driver.manage().window().maximize();
    driver.get("https://wizzair.com/en-gb/flights/timetable/clujnapoca/vienna--#/1/0/1/0/0/2019-01/2019-01");

Answer 1

您可以尝试更改 HTTP 请求中的 header ，或者更改某些系统属性以隐藏您正在使用 selenium 浏览器的事实。

设置系统属性:

System.setProperty(property, value);

要查看您的 JVM 支持哪些属性，请打开控制台(CMD/终端)并输入:

java -XshowSettings:all

设置系统属性示例:

static {
    System.setProperty("user.dir", "C:\\Users\\YourName");
}

系统属性引用: https://docs.oracle.com/javase/tutorial/essential/environment/sysprop.html

<小时/>

屏蔽您与网站的连接:

注意: URLConnection#addRequestProperty 是您设置发送到网站的 header 的方式。您可以使用HTTP或HTTPS，它们位于相同或非常接近URLConnection的通用包中。

private static final String LINUX_USER_AGENT =
        "Mozilla/5.0 (X11; Linux x86_64; rv:52) Gecko/20100101 Firefox/62" + ".0";
private static final String WINDOWS_USER_AGENT =
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:62.0) " + "Gecko/20100101 Firefox/62.0";

/**
 * Masks the URL connection as a regular one would be 403 forbidden
 *
 * @param url
 *      - URL to mask and connect to
 * @return the masked url connection to the website
 *
 * @throws IOException
 */
public static InputStreamReader getMaskedInputStream(String url) throws IOException
{
    URL website = new URL(url);
    URLConnection connection = website.openConnection();
    if (System.getProperty("os.name").contains("Win"))
    {
        connection.addRequestProperty("User-Agent", WINDOWS_USER_AGENT);
    }
    else
    {
        connection.addRequestProperty("User-Agent", LINUX_USER_AGENT);
    }
    connection.addRequestProperty("Accept-Language", "en-US,en;q=0.5");
    connection.addRequestProperty("Accept-Encoding", "gzip, deflate");
    return new InputStreamReader(connection.getInputStream());
}

/**
 * Masks the URL connection as a regular one would be 403 forbidden
 *
 * @param url
 *      - URL to mask and connect to
 * @return the masked url connection to the website
 *
 * @throws IOException
 */
public static InputStreamReader getMaskedInputStream(Proxy proxy, Authenticator auth, String url) throws IOException
{
    Authenticator.setDefault(auth);
    final URL website = new URL(url);
    final URLConnection connection = website.openConnection(proxy);
    if (System.getProperty("os.name").contains("Win"))
    {
        connection.addRequestProperty("User-Agent", WINDOWS_USER_AGENT);
    }
    else
    {
        connection.addRequestProperty("User-Agent", LINUX_USER_AGENT);
    }
    connection.addRequestProperty("Accept-Language", "en-US,en;q=0.5");
    connection.addRequestProperty("Accept-Encoding", "gzip, deflate");
    return new InputStreamReader(connection.getInputStream());
}