python - 使用 python requests 库网站总是挂起

Question

我正在尝试使用 python requests 库从这个 url https://www.adidas.com/api/products/EF2302/availability?sitePath=us 获取 html

但是，每次我运行代码时，它都会在发出获取请求时挂起

header = BASE_REQUEST_HEADER
url = 'https://www.adidas.com/api/products/EF2302/availability?sitePath=us'
r = requests.get(url, headers = header)

我检查了 Chrome 中的网络选项卡并复制了所有使用的 header (包括用户代理)，因此这不是问题。我还能够在禁用 javascript 和 cookie 的情况下在 Chrome 中加载页面。

此代码在其他网站上运行良好。我根本无法从任何阿迪达斯网站(包括 https://www.adidas.com/us )得到回复。

非常感谢任何建议。

Answer 1

该网站不喜欢请求提供的默认 User-Agent 字段，将其更改为 Firefox/Chrome(我在示例中选择了 Firefox)，即可成功读取数据:

from bs4 import BeautifulSoup
import requests
import json

headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0'}
url = 'https://www.adidas.com/api/products/EF2302/availability?sitePath=us'
r = requests.get(url, headers=headers)

json_data = json.loads(r.text)
print(json.dumps(json_data, indent=4))

打印:

{
    "id": "EF2302",
    "availability_status": "PREORDER",
    "variation_list": [
        {
            "sku": "EF2302_530",
            "availability": 15,
            "availability_status": "PREORDER",
            "size": "4",
            "instock_date": "2018-08-16T00:00:00.000Z"
        },
        {
            "sku": "EF2302_550",
            "availability": 15,
            "availability_status": "PREORDER",
            "size": "5",
            "instock_date": "2018-08-16T00:00:00.000Z"
        },
        {
            "sku": "EF2302_570",
            "availability": 15,
            "availability_status": "PREORDER",
            "size": "6",
            "instock_date": "2018-08-16T00:00:00.000Z"
        },
        {
            "sku": "EF2302_590",
            "availability": 15,
            "availability_status": "PREORDER",
            "size": "7",
            "instock_date": "2018-08-16T00:00:00.000Z"
        },
        {
            "sku": "EF2302_610",
            "availability": 15,
            "availability_status": "PREORDER",
            "size": "8",
            "instock_date": "2018-08-16T00:00:00.000Z"
        },
        {
            "sku": "EF2302_630",
            "availability": 15,
            "availability_status": "PREORDER",
            "size": "9",
            "instock_date": "2018-08-16T00:00:00.000Z"
        },
        {
            "sku": "EF2302_650",
            "availability": 15,
            "availability_status": "PREORDER",
            "size": "10",
            "instock_date": "2018-08-16T00:00:00.000Z"
        },
        {
            "sku": "EF2302_670",
            "availability": 15,
            "availability_status": "PREORDER",
            "size": "11",
            "instock_date": "2018-08-16T00:00:00.000Z"
        },
        {
            "sku": "EF2302_690",
            "availability": 15,
            "availability_status": "PREORDER",
            "size": "12",
            "instock_date": "2018-08-16T00:00:00.000Z"
        },
        {
            "sku": "EF2302_710",
            "availability": 15,
            "availability_status": "PREORDER",
            "size": "13",
            "instock_date": "2018-08-16T00:00:00.000Z"
        }
    ]
}