python - 使用 BeautifulSoup 抓取网页链接标题和 URL

Question

我有一个热门文章的网页，我想为每个引用网页的超链接和它显示的文章标题抓取。

我的脚本所需的输出是一个 CSV 文件，其中在一行中列出了每个标题和文章内容。所以如果这个网页上有 50 篇文章，我想要一个包含 50 行和 100 个数据点的文件。

我的问题是文章标题及其超链接包含在 SVG 容器中，这让我很不爽。我以前使用 BeautifulSoup 进行网页抓取，但不确定如何选择每篇文章的标题和超链接。非常感谢任何帮助。

import requests 
from bs4 import BeautifulSoup 
import re 

res = requests.get('http://fundersandfounders.com/what-internet-thinks-based-on-media/') 
res.raise_for_status() 
playFile = open('top_articles.html', 'wb') 
for chunk in res.iter_content(100000): 
    playFile.write(chunk) 
    f = open('top_articles.html') 
    soup = BeautifulSoup(f, 'html.parser') 
    links = soup.select('p') #i know this is where i'm messing up, but i'm not sure which selector to actually utilize so I'm using the paragraph selector as a place-holder
    print(links)

我知道这实际上是一个两步走的项目:我的脚本的当前版本不会遍历我要抓取其实际内容的所有超链接的列表。这是第二步，我可以自己轻松执行，但是如果有人也想写那一点，我很荣幸。

Answer 1

您应该分两步完成:

• 解析 HTML 并提取链接到 svg
• 下载svg页面，用BeautifulSoup解析它并提取“气泡”

实现:

from urllib.parse import urljoin  # Python3

import requests
from bs4 import BeautifulSoup


base_url = 'http://fundersandfounders.com/what-internet-thinks-based-on-media/'

with requests.Session() as session:
    # extract the link to svg
    res = session.get(base_url)
    soup = BeautifulSoup(res.content, 'html.parser')
    svg = soup.select_one("object.svg-content")
    svg_link = urljoin(base_url, svg["data"])

    # download and parse svg
    res = session.get(svg_link)
    soup = BeautifulSoup(res.content, 'html.parser')
    for article in soup.select("#bubbles .bgroup"):
        title, resource = [item.get_text(strip=True, separator=" ") for item in article.select("a text")]
        print("Title: '%s'; Resource: '%s'." % (title, resource))

打印文章标题和资源:

Title: 'CNET'; Resource: 'Android Apps That Extend Battery Life'.
Title: '5-Years-Old Shoots Sister'; Resource: 'CNN'.
Title: 'Samsung Galaxy Note II'; Resource: 'Engaget'.
...
Title: 'Predicting If a Couple Stays Together'; Resource: 'The Atlantic Magazine'.
Title: 'Why Doctors Die Differently'; Resource: 'The Wall Street Journal'.
Title: 'The Ideal Nap Length'; Resource: 'Lifehacker'.