python - 用 BeautifulSoup 刮掉多页

Question

我正在尝试抓取一个网址的多个页面。

但是我只能抓取第一页，因为有办法获取所有页面。

这是我的代码。

from bs4 import BeautifulSoup as Soup 
import urllib, requests, re, pandas as pd

pd.set_option('max_colwidth',500)    # to remove column limit (Otherwise, we'll lose some info) 
df = pd.DataFrame()

Comp_urls = ['https://www.indeed.com/jobs?q=Dell&rbc=DELL&jcid=0918a251e6902f97', 'https://www.indeed.com/jobs?q=Harman&rbc=Harman&jcid=4faf342d2307e9ed','https://www.indeed.com/jobs?q=johnson+%26+johnson&rbc=Johnson+%26+Johnson+Family+of+Companies&jcid=08849387e791ebc6','https://www.indeed.com/jobs?q=nova&rbc=Nova+Biomedical&jcid=051380d3bdd5b915']

for url in Comp_urls: 
    target = Soup(urllib.request.urlopen(url), "lxml")
    targetElements = target.findAll('div', class_ =' row result')

    for elem in targetElements:
        comp_name = elem.find('span', attrs={'class':'company'}).getText().strip()
        job_title = elem.find('a', attrs={'class':'turnstileLink'}).attrs['title']
        home_url = "http://www.indeed.com"
        job_link = "%s%s" % (home_url,elem.find('a').get('href'))
        job_addr = elem.find('span', attrs={'class':'location'}).getText()
        date_posted = elem.find('span', attrs={'class': 'date'}).getText()
        description = elem.find('span', attrs={'class': 'summary'}).getText().strip()


        comp_link_overall = elem.find('span', attrs={'class':'company'}).find('a')
        if comp_link_overall != None:
        comp_link_overall = "%s%s" % (home_url, comp_link_overall.attrs['href'])
        else: comp_link_overall = None

        df = df.append({'comp_name': comp_name, 'job_title': job_title,
                'job_link': job_link, 'date_posted': date_posted,
                'overall_link': comp_link_overall, 'job_location': job_addr, 'description': description
                }, ignore_index=True)


df

df.to_csv('path\\web_scrape_Indeed.csv', sep=',', encoding='utf-8')

如果有的话请提出。

Answer 1

情况 1:此处提供的代码正是您所拥有的

Comp_urls = ['https://www.indeed.com/jobs?q=Dell&rbc=DELL&jcid=0918a251e6902f97', 'https://www.indeed.com/jobs?q=Harman&rbc=Harman&jcid=4faf342d2307e9ed','https://www.indeed.com/jobs?q=johnson+%26+johnson&rbc=Johnson+%26+Johnson+Family+of+Companies&jcid=08849387e791ebc6','https://www.indeed.com/jobs?q=nova&rbc=Nova+Biomedical&jcid=051380d3bdd5b915']

for url in Comp_urls: 
    target = Soup(urllib.request.urlopen(url), "lxml")
    targetElements = target.findAll('div', class_ =' row result')

for elem in targetElements:

这里的问题是 targetElements 随着第一个 for 循环中的每次迭代而变化。

为了避免这种情况，请在第一个 for 循环内缩进第二个 for 循环，如下所示:

for url in Comp_urls: 
    target = Soup(urllib.request.urlopen(url), "lxml")
    targetElements = target.findAll('div', class_ =' row result')

    for elem in targetElements:

情况 2:您的错误不是由于缩进不当造成的 (即与您原始帖子中的内容不同) 如果您的代码被正确识别，那么 targetElements 可能是一个空列表。这意味着 target.findAll('div', class_ =' row result') 不会返回任何内容。在这种情况下，请访问站点，检查 dom，然后修改您的抓取程序。