python - Beautifulsoup - 抓取网页 - 动态加载页面

Question

我想抓取一个网页:https://www.justdial.com/Mumbai/Dairy-Product-Retailers-in-Thane/nct-10152687

我需要所有商店名称、电话号码及其地址的数据

但我最多只能做到10 导致加载您需要滚动网页的其他项目

我的代码:

import requests
import bs4

crawl_url = requests.get('https://www.justdial.com/Mumbai/Dairy-Product-
Retailers-in-Thane/nct-10152687', headers={'User-Agent': 'Mozilla/5.0'})
crawl_url.raise_for_status()


soup = bs4.BeautifulSoup(crawl_url.text, 'lxml')

for elems in soup.find_all('span', class_="jcn"):
    select_a = elems.select('a')
    for links in select_a:
        href = links.get('href')
        res = requests.get(href, headers={'User-Agent': 'Mozilla/5.0'})

        xsoup = bs4.BeautifulSoup(res.text, 'lxml')

        Name = xsoup.select('.fn')
        tel = xsoup.select('.tel')
        add = xsoup.select('.adrstxtr')
        a = Name[0]
        b = tel[0]
        c = add[0]
        print(a.getText())
        print("--"*10)
        print(b.getText())
        print("--"*10)
        print(c.getText())
        print("=="*25)

当我们向下滚动页面时，其他项目会加载 所以我想知道如何获得我想要的任意数量的数据/项目

我试过了 this

但是我并没有安静地理解好吧，而且我也没有得到那个 POST 方法:/

如果需要更多信息告诉我

Answer 1

t.m.adam 给出的解决方案有效 这是代码

import requests
import bs4

def spider(max_pages):
    page = 1
    while page < max_pages:
        url = "https://www.justdial.com/Mumbai/Dairy-Product-Retailers-in-
Thane/nct-10152687/page-%s" % page
        crawl_url = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'})
        crawl_url.raise_for_status()
        soup = bs4.BeautifulSoup(crawl_url.text, 'lxml')
        for elems in soup.find_all('span', class_="jcn"):
            select_a = elems.select('a')
            for links in select_a:
               href = links.get('href')
               res = requests.get(href, headers={'User-Agent': 
'Mozilla/5.0'})
                xsoup = bs4.BeautifulSoup(res.text, 'lxml')
                Name = xsoup.select('.fn')
                tel = xsoup.select('.tel')
                add = xsoup.select('.adrstxtr')
                a = Name[0]
                b = tel[0]
                c = add[0]
                print(a.getText())
                print("--"*10)
                print(b.getText())
                print("--"*10)
                print(c.getText())
                print("=="*25)
        page += 1


spider(3)