python - 如何使用 Python 使用 Selenium 抓取 Linkedin 上的“下一步”按钮？

Question

我正在尝试使用 Selenium 抓取 LinkedIn 网站。我无法解析“下一步”按钮。它会尽可能地抵抗。我花了半天的时间来解决这个问题，但都是徒劳。

我尝试了各种选项，包括文本等等。仅适用于开始 ID，但抓取其他按钮。

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//button[@aria-label='Далее']"} 

这对于这个网站来说很常见:

//*[starts-with(@id,'e')]

我的代码:

from selenium import webdriver
from selenium.webdriver import Keys
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from time import sleep



chrome_driver_path = Service("E:\programming\chromedriver_win32\chromedriver.exe")
driver = webdriver.Chrome(service=chrome_driver_path)
url = "https://www.linkedin.com/feed/"
driver.get(url)
SEARCH_QUERY = "python developer"
LOGIN = "EMAIL"
PASSWORD = "PASSWORD"
sleep(10)

sign_in_link = driver.find_element(By.XPATH, '/html/body/div[1]/main/p[1]/a')
sign_in_link.click()

login_input = driver.find_element(By.XPATH, '//*[@id="username"]')
login_input.send_keys(LOGIN)
sleep(1)
password_input = driver.find_element(By.XPATH, '//*[@id="password"]')
password_input.send_keys(PASSWORD)
sleep(1)
enter_button = driver.find_element(By.XPATH, '//*[@id="organic-div"]/form/div[3]/button')
enter_button.click()
sleep(25)

lens_button = driver.find_element(By.XPATH, '//*[@id="global-nav-search"]/div/button')
lens_button.click()
sleep(5)

search_input = driver.find_element(By.XPATH, '//*[@id="global-nav-typeahead"]/input')
search_input.send_keys(SEARCH_QUERY)
search_input.send_keys(Keys.ENTER)
sleep(5)

people_button = driver.find_element(By.XPATH, '//*[@id="search-reusables__filters-bar"]/ul/li[1]/button')
people_button.click()
sleep(5)

page_button = driver.find_element(By.XPATH, "//button[@aria-label='Далее']")
page_button.click()

sleep(60)

Chrome inspection of button Next Button

Answer 1

好的，这里有几个问题:

1. 代码不起作用的主要问题是因为在滚动页面之前，最初甚至不会在页面上创建“下一个”分页，因此我添加了滚动页面的机制，直到可以单击该按钮。
2. 基于本地语言文本创建定位器并不好。
3. 您应该使用WebDriverWait expected_conditions显式等待，而不是硬编码暂停。

我使用混合定位器类型来表明有时最好使用 By.ID有时By.XPATH等等
以下代码有效:

import time

from selenium import webdriver
from selenium.webdriver import Keys
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

options = Options()
options.add_argument("start-maximized")

webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 10)

url = "https://www.linkedin.com/feed/"
driver.get(url)

wait.until(EC.element_to_be_clickable((By.XPATH, "//a[contains(@href,'login')]"))).click()
wait.until(EC.element_to_be_clickable((By.ID, "username"))).send_keys(my_email)
wait.until(EC.element_to_be_clickable((By.ID, "password"))).send_keys(my_password)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[type='submit']"))).click()
search_input = wait.until(EC.element_to_be_clickable((By.XPATH, "//input[contains(@class,'search-global')]")))
search_input.click()
search_input.send_keys("python developer" + Keys.ENTER)
wait.until(EC.element_to_be_clickable((By.XPATH, '//*[@id="search-reusables__filters-bar"]/ul/li[1]/button'))).click()
wait = WebDriverWait(driver, 4)
while True:
    try:
        next_btn = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "button.artdeco-pagination__button.artdeco-pagination__button--next")))
        next_btn.location_once_scrolled_into_view
        time.sleep(0.2)
        next_btn.click()
        break
    except:
        driver.execute_script("window.scrollBy(0, arguments[0]);", 600)