python - 有没有办法使用 BeautifulSoup 将数据从列表正确转换为 CSV 文件？

Question

我正在尝试为网站创建网络爬虫。问题是收集的数据存储在列表中后，我无法将其正确写入 csv 文件。我被这个问题困扰了很长时间，希望有人知道如何解决这个问题!

从网页获取数据的循环:

import csv
from htmlrequest import simple_get
from htmlrequest import BeautifulSoup

# Define variables
listData = ['Companies', 'Locations', 'Descriptions']

plus = 15
max = 30
count = 0

# while loop to repeat process till max is reached
while (count <= max):
        start = 'https://www.companiesintheuk.co.uk/find?q=Activities+of+sport+clubs&start=' + str(count) + '&s=h&t=SicCodeSearch&location=&sicCode=93120'
        raw_html = simple_get(start)
        soup = BeautifulSoup(raw_html, 'html.parser')
        for i, div in enumerate(soup.find_all('div', class_="search_result_title")):
            listData[0] = listData[0].strip() + div.text 

        for i, div2 in enumerate(soup.find_all('div', class_="searchAddress")):
             listData[1] = listData[1].strip() + div2.text  

# This is extra information
#        for i, div3 in enumerate(soup.find_all('div', class_="searchSicCode")):
#            listData[2] = listData[2].strip() + div3.text 

        count = count + plus

打印输出示例:

Companies
(AMG) AGILITY MANAGEMENT GROUP LTD
(KLA) LIONS/LIONESS FOOTBALL TEAMS WORLD CUP LTD
(Dissolved)
1 SPORT ORGANISATION LIMITED
100UK LTD
1066 GYMNASTICS
1066 SPECIALS
10COACHING LIMITED
147 LOUNGE LTD
147 SNOOKER AND POOL CLUB (LEICESTER) LIMITED

Locations
ENGLAND, BH8 9PS
LONDON, EC2M 2PL
ENGLAND, LS7 3JB
ENGLAND, LE2 8FN
UNITED KINGDOM, N18 2QX
AVON, BS5 0JH
UNITED KINGDOM, WC2H 9JQ
UNITED KINGDOM, SE18 5SZ
UNITED KINGDOM, EC1V 2NX

我曾尝试使用此代码将其放入 CSV 文件中，但我不知道如何正确格式化我的输出!欢迎提出任何建议。

# writing to csv
with open('test.csv', 'w') as csvfile:
            write = csv.writer(csvfile, delimiter=',')
            write.writerow(['Name','Location'])
            write.writerow([listData[0],listData[1]])

print("Writing has been done!")

我希望代码能够在 csv 文件中正确格式化它，以便能够将两行导入数据库中。

This is the output when I write the data on 'test.csv'

which will result into this when opened up

The expected outcome would be something like this!

Answer 1

我不确定它是如何格式化不正确的，但也许你只需要将 with open('test.csv', 'w') 替换为 with open('test .csv', 'w+', newline='')

我已经合并了你的代码(为 requests 和 bs4 模块取出了 htmlrequests 并且也没有使用 listData，而是创建我自己的列表。我已经离开了你的列表，但它们什么都不做):

import csv
import bs4
import requests

# Define variables
listData = ['Companies', 'Locations', 'Descriptions']
company_list = []
locations_list = []
plus = 15
max = 30
count = 0

# while loop to repeat process till max is reached

while count <= max:
    start = 'https://www.companiesintheuk.co.uk/find?q=Activities+of+sport+clubs&start={}&s=h&t=SicCodeSearch&location=&sicCode=93120'.format(count)
    res = requests.get(start)
    soup = bs4.BeautifulSoup(res.text, 'html.parser')
    for i, div in enumerate(soup.find_all('div', class_="search_result_title")):
        listData[0] = listData[0].strip() + div.text
        company_list.append(div.text.strip())
    for i, div2 in enumerate(soup.find_all('div', class_="searchAddress")):
        listData[1] = listData[1].strip() + div2.text
        locations_list.append(div2.text.strip())
# This is extra information
#        for i, div3 in enumerate(soup.find_all('div', class_="searchSicCode")):
#            listData[2] = listData[2].strip() + div3.text
    count = count + plus

if len(company_list) == len(locations_list):
    with open('test.csv', 'w+', newline='') as csvfile:
        writer = csv.writer(csvfile, delimiter=',')
        writer.writerow(['Name', 'Location'])
        for i in range(len(company_list)):
            writer.writerow([company_list[i], locations_list[i]])

生成的 csv 文件如下:

Name,Location
(AMG) AGILITY MANAGEMENT GROUP LTD,"UNITED KINGDOM, M6 6DE"
"(KLA) LIONS/LIONESS FOOTBALL TEAMS WORLD CUP LTD
(Dissolved)","ENGLAND, BD1 2PX"
0161 STUDIOS LTD,"UNITED KINGDOM, HD6 3AX"
1 CLICK SPORTS MANAGEMENT LIMITED,"ENGLAND, E10 5PW"
1 SPORT ORGANISATION LIMITED,"UNITED KINGDOM, CR2 6NF"
100UK LTD,"UNITED KINGDOM, BN14 9EJ"
1066 GYMNASTICS,"EAST SUSSEX, BN21 4PT"
1066 SPECIALS,"EAST SUSSEX, TN40 1HE"
10COACHING LIMITED,"UNITED KINGDOM, SW6 6LR"
10IS ACADEMY LIMITED,"ENGLAND, PE15 9PS"
"10TH MAN LIMITED
(Dissolved)","GLASGOW, G3 6AN"
12 GAUGE EAST MANCHESTER COMMUNITY MMA LTD,"ENGLAND, OL9 8DQ"
121 MAKING WAVES LIMITED,"TYNE AND WEAR, NE30 1AR"
121 WAVES LTD,"TYNE AND WEAR, NE30 1AR"
1-2-KICK LTD,"ENGLAND, BH8 9PS"
"147 HAVANA LIMITED
(Liquidation)","LONDON, EC2M 2PL"
147 LOUNGE LTD,"ENGLAND, LS7 3JB"
147 SNOOKER AND POOL CLUB (LEICESTER) LIMITED,"ENGLAND, LE2 8FN"
1ACTIVE LTD,"UNITED KINGDOM, N18 2QX"
1ON1 KING LTD,"AVON, BS5 0JH"
1PUTT LTD,"UNITED KINGDOM, WC2H 9JQ"
1ST SPORTS LTD,"UNITED KINGDOM, SE18 5SZ"
2 BRO PRO EVENTS LTD,"UNITED KINGDOM, EC1V 2NX"
2 SPLASH SWIM SCHOOL LTD,"ENGLAND, B36 0EY"
2 STEPPERS C.I.C.,"SURREY, CR0 6BX"
2017 MOTO LIMITED,"UNITED KINGDOM, ME2 4NW"
2020 ARCHERY LTD,"LONDON, SE16 6SS"
21 LEISURE LIMITED,"LONDON, EC4M 7WS"
261 FEARLESS CLUB UNITED KINGDOM CIC,"LANCASHIRE, LA2 8RF"
2AIM4 LIMITED,"HERTFORDSHIRE, SG2 0JD"
2POINT4 FM LTD,"LONDON, NW10 8LW"
3 LIONS SCHOOL OF SPORT LTD,"BRISTOL, BS20 8BU"
3 PT LTD,"ANTRIM, BT40 2FB"
3 PUTT LIFE LTD,"UNITED KINGDOM, LU3 2DP"
3 THIRTY SEVEN LTD,"KENT, DA9 9RS"
3:30 SOCCER SCHOOL LTD,"UNITED KINGDOM, EH6 7JB"
30 MINUTE WORKOUT (LLANISHEN) LTD,"PONTYCLUN, CF72 9UA"
321 RELAX LTD,"MID GLAMORGAN, CF83 3HL"
360 MOTOR RACING CLUB LTD,"HALSTEAD, CO9 2ET"
3LIONSATHLETICS LIMITED,"ENGLAND, S3 8DB"
3S SWIM ROMFORD LTD,"UNITED KINGDOM, DA9 9DR"
3XL EVENT MANAGEMENT LIMITED,"KENT, BR3 4NW"
3XL MOTORSPORT MANAGEMENT LIMITED,"KENT, BR3 4NW"
4 CORNER FOOTBALL LTD,"BROMLEY, BR1 5DD"
4 PRO LTD,"UNITED KINGDOM, FY5 5HT"

这对我来说似乎很好，但是您的帖子不清楚您希望它如何格式化，所以我真的不知道