python - Selenium Python = 在无限滚动中单击按钮 "ShowMore"时出现问题

Question

我正在抓取 https://play.google.com/ 中的 Mobile Legend 评论数据。我希望我的机器人能够自行向下滚动并尽可能多地加载一些评论。在机器人完成我想要的操作后，该机器人抓取所有评论。

The Problem is when the bot do the infinite Scroll down and click the "Showmore" Button, somehow the second click of "Showmore" button gived me error ([7200:8128:0903/172837.024:ERROR:gpu_init.cc(441)] Passthrough is not supported, GL is disabled) and the looping is break.

from selenium import webdriver
from time import sleep
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.webdriver.chrome.options import Options
import pandas as pd

#activate GL first
options = Options()
#options.add_argument("--kiosk")#fullscreen
options.add_argument('--enable-webgl-draft-extensions')
driver = webdriver.Chrome('D:\chromedriver', chrome_options = options)
driver.maximize_window()
print("WebGL Activated")

#open google play
driver.get("https://play.google.com/store/apps/details?id=com.mobile.legends&showAllReviews=true")
sleep(10)
action = ActionChains(driver)

# Get scroll height
last_height = driver.execute_script("return document.body.scrollHeight")
SCROLL_PAUSE_TIME = 10

#this variable limit the infinite looping
click = 0

while not(click == 100):
    # Scroll down to bottom
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

    # Wait to load page
    sleep(SCROLL_PAUSE_TIME)

    # Calculate new scroll height and compare with last scroll height
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        try:
            sleep(7)
            showMore = driver.find_element_by_class_name("U26fgb.O0WRkf.oG5Srb.C0oVfc.n9lfJ.M9Bg4d")
            action.move_to_element(showMore)
            action.click(showMore)
            action.perform()
            sleep(10)            
            print("Click Showmore "+str(click))
            click += 1
        except: 
            print("------Scroll Finish-------")
            print("Click ShowMore Counts = " + str(click))
            break
    last_height = new_height

这是终端的输出:

WebGL Activated
[5296:3760:0903/174720.883:ERROR:device_event_log_impl.cc(214)] [17:47:20.883] USB: usb_device_handle_win.cc:1048 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)
[5296:3760:0903/174720.889:ERROR:device_event_log_impl.cc(214)] [17:47:20.889] Bluetooth: bluetooth_adapter_winrt.cc:713 GetBluetoothAdapterStaticsActivationFactory failed: Class not registered (0x80040154)
Click Showmore 0
[9052:7864:0903/174913.082:ERROR:gpu_init.cc(441)] Passthrough is not supported, GL is disabled
------Scroll Finish-------
Click ShowMore Counts = 1

Answer 1

这是我的方法。使用 END 键 8 次并单击“显示更多”...评论太多...如果您需要任何帮助抓取评论，请告诉我。

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains
from selenium.common.exceptions import NoSuchElementException
import time
url = "https://play.google.com/store/apps/details?id=com.mobile.legends&showAllReviews=true"

d = webdriver.Chrome(ChromeDriverManager().install())
d.get(url)
while True:
    for _ in range(8):
        actions = ActionChains(d)
        actions.send_keys(Keys.END).perform()
        time.sleep(1)
    try:
        d.find_element_by_class_name("CwaK9").click()
    except NoSuchElementException:
        break