python - 如果表 beautifulsoup 和 pandas 中不存在类，则停止抓取 url

Question

我正在使用 csv 文件中的 url 列表从 html 表中抓取和提取数据。当表中不存在“style3”时，我想停止浏览 url。 我创建了一个函数，如果它不存在，它将返回 false，但我对如何实际实现它感到困惑。

任何关于解决方案的建议或对文献的指导都会有很大帮助，因为我在这里找不到任何可以帮助我弄清楚的东西。

我添加了 1 个带有“style3”的 url 和 1 个不带“style3”的 url。感谢您提供的所有帮助。

http://www.wvlabor.com/new_searches/contractor_RESULTS.cfm?wvnumber=WV057808&contractor_name=&dba=&city_name=&County=&Submit3=Search+Contractors http://www.wvlabor.com/new_searches/contractor_RESULTS.cfm?wvnumber=WV057924&contractor_name=&dba=&city_name=&County=&Submit3=Search+Contractors

import csv
from urllib.request import urlopen
import pandas as pd
from bs4 import BeautifulSoup as BS

def license_exists(soup):
    contents = []
    with open('WV_urls.csv','r') as csvf:
        urls = csv.reader(csvf)
        for url in urls:
            if soup(class_='style3'):
                return True
            else:
                return False

contents = []
more = True
while more:
    df  = pd.DataFrame(columns=['WV Number', 'Company', 'DBA', 'Address', 'City', 'State', 'Zip','County', 'Phone', 'Classification*', 'Expires']) #initialize the data frame with columns
    with open('WV_urls.csv','r') as csvf: # Open file in read mode
        urls = csv.reader(csvf)
        for url in urls:
            contents.append(url) # Add each url to list contents
        for url in contents:  # Parse through each url in the list.
            page = urlopen(url[0]).read()
            df1, header = pd.read_html(page,header=0)#reading with header
            more = license_exists(?????)
            df=df.append(df1) # append to dataframe

            df.to_csv('WV_Licenses_Daily.csv', index=False)

Answer 1

您可以使用单个 for 循环和中断来完成此操作(不需要 while more):

lst = []
with open('WV_urls.csv','r') as csvf: # Open file in read mode
    urls = csv.reader(csvf)
    for url in urls:
        page = urlopen(url[0]).read()
        df1, header = pd.read_html(page, header=0)
        if license_exists(BS(page, ‘html.parser’)):
            # if the license is present we don't want to parse any more urls.
            # Note: we don't append this last result (should we?)
            break
        lst.append(df1)

df = pd.concat(lst)
df.to_csv('WV_Licenses_Daily.csv', index=False)

注意:这会从 DataFrame 列表中创建最终的 DataFrame，这比每次附加都更有效。