python - 如何使用 Selenium、Python 从 Google 搜索中提取链接

Question

我试图让 Google 提取查询的相关搜索链接，在本例中我使用的是维基百科，然后通过 Selenium 解析前三个的 url。到目前为止，我只能完成第一部分，谷歌搜索。这是我的代码:

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait # available since 2.4.0
from selenium.webdriver.support import expected_conditions as EC # available since 2.26.0

query = raw_input("What do you wish to search on Wikipedia?\n")
query = " " + query

# Create a new instance of the Firefox driver
driver = webdriver.Firefox()

# go to the google home page
driver.get("https://www.google.com/search?q=site%3Awikipedia.com&ie=utf-8&oe=utf-8")

# the page is ajaxy so the title is originally this:
print driver.title

# find the element that's name attribute is q (the google search box)
inputElement = driver.find_element_by_name("q")

# type in the search
inputElement.send_keys(query)

# submit the form (although google automatically searches now without submitting)
inputElement.submit()

try:
    # we have to wait for the page to refresh, the last thing that seems to be updated is the title

    # You should see "cheese! - Google Search"
    print driver.title

    driver.find_element_by_xpath("//h3[contains(text(),'Wikipedia')]").click()

finally:
    driver.quit()

我正在尝试使用 Selenium 文档中的示例，所以请原谅评论和有时不必要的代码。

我遇到问题的代码行是:

driver.find_element_by_xpath("//h3[contains(text(),'Wikipedia')]").click()

我试图做的是获取相关的维基百科链接，或者更具体地说，是 H3“r”路径指向的链接。

Here's a picture of a Google page that I'm describing.

在这种情况下，我希望拉取链接 http://en.wikipedia.com/wiki/salary

对不起，文字太长了，我尽量说得具体些。无论如何，感谢您提前提供的帮助。

最好的问候!

Answer 1

问题是这个 XPath 不正确 - 文本中有一个 a 元素，而不是 h3 元素。修复它:

driver.find_element_by_xpath("//a[contains(text(), 'Wikipedia')]").click()

您甚至可以进一步简化它:

driver.find_element_by_partial_link_text("Wikipedia").click()