python-3.x - 连接多个具有相同列名的 CSV

Question

我在连接这些 pandas 数据帧时遇到了麻烦，因为我不断收到错误消息 pandas.errors.InvalidIndexError: Reindexing only valid with uniquely valued Index objects 我也试图让我的代码不那么笨拙，运行更顺畅。我还想知道是否有一种方法可以使用 python 在一个 csv 上获取多个页面。任何帮助都会很棒。

import requests
from bs4 import BeautifulSoup
import pandas as pd

headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'}

URL = "https://www.collincad.org/propertysearch?situs_street=Willowgate&situs_street_suffix" \
      "=&isd%5B%5D=any&city%5B%5D=any&prop_type%5B%5D=R&prop_type%5B%5D=P&prop_type%5B%5D=MH&active%5B%5D=1&year=2021&sort=G&page_number=1"

t = URL + "&page_number="
URL2 = t + "2"
URL3 = t + "3"

s = requests.Session()

data = []

page = s.get(URL,headers=headers)
page2 = s.get(URL2, headers=headers)
page3 = s.get(URL3, headers=headers)

soup = BeautifulSoup(page.content, "lxml")
soup2 = BeautifulSoup(page2.content, "lxml")
soup3 = BeautifulSoup(page3.content, "lxml")


for row in soup.select('#propertysearchresults tr'):
    data.append([c.get_text(' ',strip=True) for c in row.select('td')])
for row in soup2.select('#propertysearchresults tr'):
    data.append([c.get_text(' ',strip=True) for c in row.select('td')])
for row in soup3.select('#propertysearchresults tr'):
    data.append([c.get_text(' ',strip=True) for c in row.select('td')])


df1 = pd.DataFrame(data[1:], columns=data[0])
df2 = pd.DataFrame(data[2:], columns=data[1])
df3 = pd.DataFrame(data[3:], columns=data[2])

final = pd.concat([df1, df2, df3], axis=0)

final.to_csv('Street.csv', encoding='utf-8')

Answer 1

会发生什么？

如@Zach Young所述data已经保存了您想要转换为一个数据框的所有行。所以这不是pandas的问题更多的是如何收集信息的问题。

如何修复？

基于您问题中的代码的方法是选择更具体的表数据 - 请注意 tbody在选择中，这将排除标题:

for row in soup.select('#propertysearchresults tbody tr'):
    data.append([c.get_text(' ',strip=True) for c in row.select('td')])

创建数据框时，您可以另外设置列标题:

pd.DataFrame(data, columns=[c.get_text(' ',strip=True) for c in soup.select('#propertysearchresults thead td')])

示例

这将展示如何迭代包含表格的网站的不同页面:

import requests
from bs4 import BeautifulSoup
import pandas as pd

headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'}

URL = "https://www.collincad.org/propertysearch?situs_street=Willowgate&situs_street_suffix" \
      "=&isd%5B%5D=any&city%5B%5D=any&prop_type%5B%5D=R&prop_type%5B%5D=P&prop_type%5B%5D=MH&active%5B%5D=1&year=2021&sort=G&page_number=1"

s = requests.Session()

data = []
while True:

    page = s.get(URL,headers=headers)
    soup = BeautifulSoup(page.content, "lxml")

    for row in soup.select('#propertysearchresults tbody tr'):
        data.append([c.get_text(' ',strip=True) for c in row.select('td')])

    if (a := soup.select_one('#page_selector strong + a')):
        URL = "https://www.collincad.org"+a['href']
    else:
        break


pd.DataFrame(data, columns=[c.get_text(' ',strip=True) for c in soup.select('#propertysearchresults thead td')])

输出

<表类=“s-表”> <标题> 属性 ID ↓ 地理 ID ↓ 所有者名称 特性地址 法律说明 2021 年市场值(value) <正文> 1 2709013 R-10644-00H-0010-1 PARTHASARATHY SURESH 和 ANITHA HARIKRISHNAN 12209 Willowgate Dr Frisco, TX 75035 Ridgeview At Panther Creek 2 期，H 座，地 block 1 513,019 美元 ... ... ... ... ... ... 61 2129238 R-4734-00C-0110-1 赫普·阿隆 990 Willowgate Dr Prosper, TX 75078 柳岭一期，C座，11号地段 509,795 美元