我试图从网页中获取不同公司的链接，但我尝试使用的脚本抛出以下错误。在 chrome 开发工具中，我可以看到我可以使用 post http 请求获取不同公司的 ids。但是，如果我可以获得 ids，那么我将能够使用此链接 'https://angel.co/startups/{}' 添加 id字符串格式以创建完整的公司链接。
我试过:
import requests
link = 'https://angel.co/company_filters/search_data'
base = 'https://angel.co/startups/{}'
payload={'sort':'signal','page':'2'}
r = requests.post(link,data=payload,headers={
'x-requested-with':'XMLHttpRequest'
'User-Agent":"Mozilla/5.0'
})
print(r.json())
上面的脚本抛出以下错误:
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
如何使用请求从上述站点获取不同公司的链接？
最佳答案
我制作了函数 get_soup(page)，它接受来自 1 的 page 参数并返回带有相关数据的 soup。您可以将此函数放入循环中以抓取更多页面:
import requests
from bs4 import BeautifulSoup
def get_soup(page=1):
headers = {
'Accept-Language' : 'en-US,en;q=0.5',
'Host' : 'angel.co',
'User-Agent' : 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0'
}
payload={'sort':'signal','page':str(page)}
url = 'https://angel.co/company_filters/search_data'
data = requests.get(url, headers=headers, data=payload).json()
new_url = 'https://angel.co/companies/startups?' + '&'.join('ids[]={}'.format(_id) for _id in data['ids'])
new_url += '&sort=' + data['sort']
new_url += '&total=' + str(data['total'])
new_url += '&page=' + str(data['page'])
new_url += '&new=' + str(data['new']).lower()
new_url += '&hexdigest=' + data['hexdigest']
data = requests.get(new_url, headers=headers).json()
return BeautifulSoup(data['html'], 'lxml')
soup = get_soup(1)
rows = []
for company, joined, location, market, website, company_size, stage, raised in zip(soup.select('.column.company'),
soup.select('.column.joined .value'),
soup.select('.column.location .value'),
soup.select('.column.market .value'),
soup.select('.column.website .value'),
soup.select('.column.company_size .value'),
soup.select('.column.stage .value'),
soup.select('.column.raised .value')):
company = company.get_text(strip=True, separator=" ")
joined = joined.get_text(strip=True)
location = location.get_text(strip=True)
market = market.get_text(strip=True)
website = website.get_text(strip=True)
company_size = company_size.get_text(strip=True)
stage = stage.get_text(strip=True)
raised = raised.get_text(strip=True)
rows.append([company, joined, location, market, website, company_size, stage, raised])
from textwrap import shorten
print(''.join('{: <25}'.format(shorten(d, 25)) for d in ['Company', 'Joined', 'Location', 'Market', 'Website', 'Company Size', 'Stage', 'Raised']))
print('-' * (25*8))
for row in rows:
print(''.join('{: <25}'.format(shorten(d, 25)) for d in row))
打印:
Company Joined Location Market Website Company Size Stage Raised
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Nutanix Your [...] May ’14 San Jose Virtualization nutanix.com 1001-5000 IPO $312,200,000
EverFi Oct ’12 Washington DC Education everfi.com 51-200 Series C $61,000,000
Butter Make friends [...]Jun ’14 San Francisco Messaging getbutter.me 1-10 Seed $371,500
Fluent The future [...] Mar ’12 Sydney Curated Web fluent.io - - -
Belly Sep ’12 Chicago Small and Medium [...] bellycard.com Series B $24,975,000
Autotech Ventures [...] Apr ’14 Menlo Park Internet of Things autotechvc.com 1-10 - -
Oscar Health [...] Jun ’14 Tempe Technology hioscar.com 1001-5000 $1,267,500,000
Tovala Smart oven [...] Feb ’16 Chicago Home Automation tovala.com 11-50 Series A $10,800,000
GiftRocket Online [...] Mar ’16 San Francisco Gift Card giftrocket.com 1-10 Seed $520,000
Elemeno Health B2B [...] Apr ’16 Oakland Training elemenohealth.com 1-10 Seed $1,635,000
Sudo Technologies [...] Apr ’16 Menlo Park - sudo.ai - -
Stypi Sep ’16 - - Acquired -
Amazon Alexa Amazon [...]Sep ’16 Cambridge Speech Recognition developer.amazon.com 11-50 - -
Altos Ventures A [...] Oct ’16 Menlo Park Technology altos.vc 1-10 - -
Flirtey Making [...] Oct ’16 Reno - flirtey.com 11-50 Series A $16,000,000
SV Liquidity Fund [...] Oct ’16 San Francisco B2B svlq.io 1-10 - -
Princeton Ventures [...] Jan ’17 Princeton Technology princetonventures.com 1-10 - -
hulu - Beijing [...] Jan ’17 Beijing TV Production hulu.com - - -
Distributed Systems [...]Jan ’17 San Francisco Identity pavlov.ai 1-10 - -
Fetch Marketplace [...] May ’17 Atlanta Technology fetchtruck.com 1-10 Seed -
编辑:为了获得链接，您可以:
soup = get_soup(1)
for a in soup.select('.website a[href]'):
print(a['href'])
打印:
http://www.fuelpowered.com
http://www.slide.com
http://www.mparticle.com
http://www.matter.io
http://www.smartling.com
https://stensul.com
https://avametric.com/
https://ledgerinvesting.com
http://www.relativityspace.com
http://teamdom.co
http://www.wonderschool.com
http://www.upcall.com
http://focal.systems
https://asktetra.com
https://www.subdreamstudios.com/
http://www.stedi.com
http://www.magnarapp.com/
http://www.kylie.ai
http://clipboardhealth.com
关于python - 无法使用请求从网站上抓取不同公司的链接，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/57242013/