python - 使用python下载bing图像搜索结果(自定义url)

Question

我想使用 python 代码下载 bing 搜索图像。

示例网址: https://www.bing.com/images/search?q=sketch%2520using%20iphone%2520students

我的 python 代码生成 bing 搜索的 url，如示例所示。下一步是将该链接中显示的所有图像下载到我的本地桌面上。

在我的项目中，我用 python 生成一些单词，我的代码生成 bing 图像搜索 URL。我需要的只是使用 python 下载该搜索页面上显示的图像。

Answer 1

要下载图像，您需要向以 .png 结尾的图像 URL 发出请求。 , .jpg等等

但是 Bing 提供了 "m" <a> 内的属性将所需数据存储在 JSON format 中的元素您可以从中解析存储在"murl"中的图像URL。 key 并随后下载。

要将所有图像下载到本地计算机，您可以使用 2 种方法:

# bs4

for index, url in enumerate(soup.select(".iusc"), start=1):
    img_url = json.loads(url["m"])["murl"]
    image = requests.get(img_url, headers=headers, timeout=30)
    query = query.lower().replace(" ", "_")
    
    if image.status_code == 200:
        with open(f"images/{query}_image_{index}.jpg", 'wb') as file:
            file.write(image.content)

# urllib

for index, url in enumerate(soup.select(".iusc"), start=1):
    img_url = json.loads(url["m"])["murl"]
    query = query.lower().replace(" ", "_")

    opener = req.build_opener()
    opener.addheaders=[("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.60 Safari/537.36")]
    req.install_opener(opener)
    req.urlretrieve(img_url, f"images/{query}_image_{index}.jpg")

在第一种情况下，您可以使用 context manager with open() 在本地加载图像。在第二种情况下，您可以使用 urllib.request.urlretrieve method urllib.request library的.

此外，请确保您使用的是 request headers user-agent 充当“真实”用户访问。因为默认requests user-agent是 python-requests 并且网站知道它很可能是发送请求的脚本。 Check what's your user-agent .

注意: urllib.request.urlretrieve 可能会出现错误某些请求已获得验证码或返回不成功的其他内容的方法 status code 。最大的问题是很难在 requests 时测试响应代码。提供 status_code 方法进行测试。

---

代码和full example in online IDE :

from bs4 import BeautifulSoup
import requests, lxml, json

query = "sketch using iphone students"

# https://docs.python-requests.org/en/master/user/quickstart/#passing-parameters-in-urls
params = {
    "q": query,
    "first": 1
}

# https://docs.python-requests.org/en/master/user/quickstart/#custom-headers
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.60 Safari/537.36"
}

response = requests.get("https://www.bing.com/images/search", params=params, headers=headers, timeout=30)
soup = BeautifulSoup(response.text, "lxml")

for index, url in enumerate(soup.select(".iusc"), start=1):
    img_url = json.loads(url["m"])["murl"]
    image = requests.get(img_url, headers=headers, timeout=30)
    query = query.lower().replace(" ", "_")
    
    if image.status_code == 200:
        with open(f"images/{query}_image_{index}.jpg", 'wb') as file:
            file.write(image.content)

---

使用 urllib.request.urlretrieve .

from bs4 import BeautifulSoup
import requests, lxml, json
import urllib.request as req

query = "sketch using iphone students"

# https://docs.python-requests.org/en/master/user/quickstart/#passing-parameters-in-urls
params = {
    "q": query,
    "first": 1
}

# https://docs.python-requests.org/en/master/user/quickstart/#custom-headers
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.60 Safari/537.36"
}

response = requests.get("https://www.bing.com/images/search", params=params, headers=headers, timeout=30)
soup = BeautifulSoup(response.text, "lxml")

for index, url in enumerate(soup.select(".iusc"), start=1):
    img_url = json.loads(url["m"])["murl"]
    query = query.lower().replace(" ", "_")

    opener = req.build_opener()
    opener.addheaders=[("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.60 Safari/537.36")]
    req.install_opener(opener)
    req.urlretrieve(img_url, f"images/{query}_image_{index}.jpg")

---

输出: