python - 无法使用请求从网页中抓取一些静态图像链接

Question

我正在尝试从网站的着陆页抓取图片。所有图像都在 search_results 内类(class)名称。当我运行下面的脚本时，我没有得到任何结果。我检查了 status_code并且可以注意到脚本正在获取 403 .

website link

由于图像是静态的并且在页面源中可用，我如何使用请求抓取图像链接？

import requests
from bs4 import BeautifulSoup

url = 'https://pixabay.com/images/search/office/'

headers = {
    'User-Agent':'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36',
}

r = requests.get(url,headers=headers)
print(r.status_code)
soup = BeautifulSoup(r.text,"lxml")
for item in soup.select(".search_results a > img[src]"):
    print(item.get("src"))

任何与任何浏览器模拟器相关的解决方案，如在 selenium 中，都不是我要找的。

Answer 1

这使用 Selenium .然而，出于某种原因，它似乎没有在 headless 模式下找到图像:

from selenium import webdriver
from bs4 import BeautifulSoup


options = webdriver.ChromeOptions()
#options.add_argument("headless")
options.add_experimental_option('excludeSwitches', ['enable-logging'])
driver = webdriver.Chrome(options=options)
try:
    driver.implicitly_wait(3)
    driver.get('https://pixabay.com/images/search/office')
    images = driver.find_elements_by_css_selector('.search_results a > img[src]') # wait for images to show up
    soup = BeautifulSoup(driver.page_source, 'lxml')
    for item in soup.select(".search_results a > img[src]"):
        print(item.get("src"))
finally:
    driver.quit()

打印:

https://cdn.pixabay.com/photo/2016/03/09/09/22/workplace-1245776__340.jpg
https://cdn.pixabay.com/photo/2015/01/08/18/26/write-593333__340.jpg
https://cdn.pixabay.com/photo/2015/02/02/11/09/office-620822__340.jpg
https://cdn.pixabay.com/photo/2014/05/02/21/50/home-office-336378__340.jpg
https://cdn.pixabay.com/photo/2016/02/19/11/19/office-1209640__340.jpg
https://cdn.pixabay.com/photo/2015/02/02/11/08/office-620817__340.jpg
https://cdn.pixabay.com/photo/2016/03/26/13/09/cup-of-coffee-1280537__340.jpg
https://cdn.pixabay.com/photo/2017/05/11/11/15/workplace-2303851__340.jpg
https://cdn.pixabay.com/photo/2015/01/09/11/08/startup-594090__340.jpg
https://cdn.pixabay.com/photo/2015/01/08/18/25/startup-593327__340.jpg
https://cdn.pixabay.com/photo/2015/01/08/18/27/startup-593341__340.jpg
https://cdn.pixabay.com/photo/2014/05/02/21/49/home-office-336373__340.jpg
https://cdn.pixabay.com/photo/2015/01/09/11/11/office-594132__340.jpg
https://cdn.pixabay.com/photo/2017/05/04/16/37/meeting-2284501__340.jpg
https://cdn.pixabay.com/photo/2014/05/03/01/03/macbook-336704__340.jpg
https://cdn.pixabay.com/photo/2018/01/11/21/27/desk-3076954__340.jpg
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif
/static/img/blank.gif