python - 如何在selenium上抓取产品详细信息页面

Question

我正在学习 Selenium 。现在我的这段代码可以从这个url https://www.daraz.com.bd/consumer-electronics/?spm=a2a0e.pdp.breadcrumb.1.4d20110bzkC0bn 的字体页面中抓取所有产品标题。但我想单击此页面的每个产品链接，这将带我进入产品详细信息页面，以便我可以从产品详细信息页面抓取信息。这是我的代码:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException

#argument for incognito Chrome
option = webdriver.ChromeOptions()
option.add_argument(" — incognito")

browser = webdriver.Chrome()

browser.get("https://www.daraz.com.bd/consumer-electronics/?spm=a2a0e.pdp.breadcrumb.1.4d20110bzkC0bn")

# Wait 20 seconds for page to load
timeout = 20
try:
    WebDriverWait(browser, timeout).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='c16H9d']")))
except TimeoutException:
    print("Timed out waiting for page to load")
    browser.quit()



# find_elements_by_xpath returns an array of selenium objects.
titles_element = browser.find_elements_by_xpath("//div[@class='c16H9d']")


# use list comprehension to get the actual repo titles and not the selenium objects.
titles = [x.text for x in titles_element]
# print out all the titles.
print('titles:')
print(titles, '\n')
browser.quit()

Answer 1

我建议您获取href并一一打开，而不是您所说的点击。

您需要此定位器:By.XPATH, "//div[@class='c16H9d']//a"，并使用 .visibility_of_all_elements_ located 等待所有元素而不是 .visibility_of_element_ located 。

之后，使用以下方法获取href:.get_attribute('href')

并打开一个新窗口，其中包含已获取的特定 href。

browser.get("https://www.daraz.com.bd/consumer-electronics/?spm=a2a0e.pdp.breadcrumb.1.4d20110bzkC0bn")

# Wait 20 seconds for page to load
timeout = 20

elements = WebDriverWait(browser, timeout).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='c16H9d']//a")))

for element in elements:
    #get href
    href = element.get_attribute('href')
    print(href)
    #open new window with specific href
    browser.execute_script("window.open('" +href +"');")
    # switch to new window
    browser.switch_to.window(browser.window_handles[1])


    #......now you are on the new window, scrape here
    #example to scrape 'title' in the new window
    xx = WebDriverWait(browser, timeout).until(EC.visibility_of_element_located((By.CLASS_NAME, "pdp-mod-product-badge-title")))
    print(xx.text)


    #close the new window
    browser.close()
    #back to main window
    browser.switch_to.window(browser.window_handles[0])

browser.quit()