我正在尝试返回公司名称列表的 URL,然后更新 pandas 数据框以包含返回的 URL。
我正在导入所有依赖项并读取 .csv 文件:
import logging
import os
import pandas as pd
import re
from googlesearch import search
df = pd.read_csv('Building_Contractors_Stephen_V1.csv')
df.head()
然后我定义一个获取所有 URL 的函数:
def get_urls(tag, n, language):
urls = [url for url in search(tag, stop=n, lang=language)][:n]
return urls
然后我在一个网址上进行测试:
test_return = get_urls(df.Hospital_Building_Contractors[0], 10, 'en')
test_return
返回 URL 列表:
['https://www.turnerconstruction.com/',
'http://www.turnerconstruction.com/careers',
'http://www.turnerconstruction.com/office-network',
'http://www.turnerconstruction.com/about-us',
'http://www.turnerconstruction.com/turner-university',
'http://www.turnerconstruction.com/careers/jobs',
'http://www.turnerconstruction.com/about-us/where-we-work',
'https://en.wikipedia.org/wiki/Turner_Construction',
'https://en.wikipedia.org/wiki/Turner_Construction#History',
'https://en.wikipedia.org/wiki/Turner_Construction#Early_years']
我似乎无法弄清楚如何迭代列表中的所有项目,并将它们添加到数据框中的新列。
这是我的代码:
i = 0
num = len(df.Hospital_Building_Contractors)
while i < num:
get_urls(df.Hospital_Building_Contractors[i], 1, 'en')
df.insert(1, "URL", urls, allow_duplicates=True)
i += 1
返回此错误:
NameError Traceback (most recent call last) in 17 while i < num: 18 get_urls(df.Hospital_Building_Contractors[i], 1, 'en') ---> 19 df.insert(1, "URL", urls, allow_duplicates=True) 20 i += 1 21
NameError: name 'urls' is not defined
我确信这是一个很容易解决的问题,但我很困惑。 我在 get_urls() 函数中定义“urls”;所以我不确定为什么会出现这个错误。
理想情况下,我会有一个像这样的解决方案:
a = 0
num = len(df.Hospital_Building_Contractors)
while a < num:
get_urls(df.Hospital_Building_Contractors[a], 1, 'en')
df.insert(1, "URL", urls, allow_duplicates=True)
a += 1
b = 0
num = len(df.University_Building_Contractors)
while b < num:
get_urls(df.University_Building_Contractors[b], 1, 'en')
df.insert(3, "URL", urls, allow_duplicates=True)
b += 1
c = 0
num = len(df.Hospital_Building_Contractors)
while c < num:
get_urls(df.Hospital_Building_Contractors[c], 1, 'en')
df.insert(5, "URL", urls, allow_duplicates=True)
c += 1
它将迭代每个列表,查找 URL 并将它们添加到数据帧中。
最佳答案
使用带有过滤器的自定义lambda 函数
,仅处理字符串:
from googlesearch import search
df = pd.read_csv('Building_Contractors_Stephen_V1.csv')
#print (df)
def get_urls(tag, n, language):
urls = [url for url in search(tag, stop=n, lang=language)][:n]
return urls
#for only one top1 value
f = lambda x: next(iter(get_urls(x, 1, 'en') if isinstance(x, str) else []), 'no value')
#for multiple top values, eg. top3
#f = lambda x: get_urls(x, 3, 'en') if isinstance(x, str) else []
df['a'] = df.Hospital_Building_Contractors.apply(f)
df['b'] = df.University_Building_Contractors.apply(f)
df['c'] = df.Military_Contractors.apply(f)
print (df.tail())
Hospital_Building_Contractors University_Building_Contractors \
104 Progressive AE NaN
105 Hellas Construction NaN
106 Wight & Company NaN
107 PWI Engineering NaN
108 Cordogan Clark & Associates NaN
Military_Contractors a \
104 NaN https://www.linkedin.com/company/progressive-ae
105 NaN http://www.hellasconstruction.com/
106 NaN https://www.wightco.com/
107 NaN http://www.pwius.com/
108 NaN http://www.cordoganclark.com/
b c
104 no value no value
105 no value no value
106 no value no value
107 no value no value
108 no value no value
关于python - 将抓取的列表添加到 Pandas Dataframe,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/54663468/