Python BeautifulSoup 获取图像、文本和 URL

Question

我正在尝试从以下网址(http://www.ancient-hebrew.org/m/dictionary/1000.html)抓取数据。所以每个希伯来语单词部分都以 img urls 开头，后跟 2 个文本，即实际的希伯来语单词及其发音。例如，url 中的第一个条目是以下“img1 img2 img3 אֶלֶף e-leph”，使用 wget 下载 html 后，希伯来语单词是一个 unicode

例如，我提供的以下代码 <img src="../../files/heb-anc-sm-pey.jpg"/>和 <font face="arial" size="+1"> unicode_hebrew_text </font>和 <a href="audio/ 505 .mp3"><img border="0" height="25" src="../../files/icon_audio.gif" width="25"/></a>

相反，我只想要 ../../files/heb-anc-sm-pey.jpg 和 unicode_hebrew_text和 audio/505.mp3 (without any spaces in between)

from bs4 import BeautifulSoup

raw_html = open('/Users/gansaikhanshur/TESTING/webScraping/1000.html').read()
html = BeautifulSoup(raw_html, 'html.parser')

# output: <img src="../../files/heb-anc-sm-pey.jpg"/>
imgs = html.findAll("img")
for image in imgs:
    # print image source
    if "jpg" in str(image):
        print(image)

# output: <font face="arial" size="+1"> unicode_hebrew_text </font>
font = html('font', face="arial", size="+1")
for f in font:
    continue

# output: <a href="audio/   505 .mp3"><img border="0" height="25" src="../../files/icon_audio.gif" width="25"/></a>
mp3file = html.findAll(href=True)
for mp3 in mp3file:
    if "mp3" in str(mp3):
        continue

如您所见，我的代码并没有真正完成这项工作。最后，我想获取 URL 中每个单词的信息，并将其保存为文本文件或 json 文件，以较容易的为准。

例如，图片:URLsOfImages，希伯来语单词:txt，发音:txt，URLtoAudio:txt

以及下一个词等等。

Answer 1

您要抓取的网页有点棘手。

现在，我为什么这么说？问题是，网页并没有大量使用 div 标签或类似的东西来将单个希伯来语单词及其相关数据放在一个 block 中(让我称希伯来语单词及其相关数据为希伯来语 block )。但是，如果您关注页面源代码，那么您会注意到每个希伯来语 block 都从一个 anchor 标记开始，看起来像这样:<a name="505"></a> .值“505”将根据每个希伯来语 block 的强数而变化。所以，我试着把它作为每个希伯来语 block 的起点。现在在希伯来语 block 中，您可以使用扬声器图标 <a href="audio/ 505 .mp3"><img src="../../files/icon_audio.gif" width="25" height="25" border="0"></a>来表示希伯来语 block 的终点。

考虑到上述想法，我想到了这个:

from bs4 import BeautifulSoup
from bs4 import NavigableString, Tag 
import requests
import re

def make_soup(url):
    html = requests.get(url, verify = False)
    soup = BeautifulSoup(html.text, 'html.parser')
    return soup


url = "http://www.ancient-hebrew.org/m/dictionary/1000.html"
soup = make_soup(url)

points = soup.find_all("a", attrs = {"name" : re.compile("[0-9]+")}) 
#This will help you locate the starting points of the Hebrew block

for point in points:
    point_name = point['name'] 
    images = [] #Will hold the links for the Old Hebrew images
    text = "" #Will hold the immediate next Unicode Hebrew text
    usage = "" #Will hold the usage/meaning data
    mp3 = "" #Will hold the link to audio file

    temp = point.next_sibling 
    #'temp' variable will be used as a navigator to extract the data within the Hebrew block 

    while(True):
        #If 'temp' is a HTML 'Tag' by nature, then it'll contain the Unicode Hebrew text and links for images& audio
        if(type(temp) is Tag): 
            if(temp.name == 'img'):        
                #If it's an 'img' tag, then put it to 'images' list 
                images.append(temp['src'])
            elif(temp.name == 'font'):
                #If it's a 'font' tag, then put it in 'text'
                text = temp.string
            elif(temp.name == 'a'):
                #If it's 'a' tag, then it contains the link to audio file
                mp3 = temp['href']  
                #Link to audio file will mark the end of the Hebrew block. So, 'break' out
                break
        elif(type(temp) is NavigableString):
            #If 'temp' acquires a 'NavigableString' instead of a 'Tag' then it's usage/meaning
            usage += temp
        #Move to the next item within the Hebrew block
        temp = temp.next_sibling

    print([point_name, images, mp3, text, usage])