python - 如何使用 python 抓取 onclick 值 (<li><a id ="FR123456"onclick=) 及其名称和图像？

Question

我得到了一个 html 字符串(responseData)，其中有很多 <li> .. </li> 集。我想从每组 <li> ...</li> 中解析以下数据:

1: call.php?category=fruits&fruitid=123456
2: mango season
3: http://imagehosting.com/images/fru_123456.png

获取上述数据后，我想对每个 url(call.php?category=fruits&fruitid=123456) 发出 http 请求，找到并获取其结果以进行进一步处理。谁能告诉我如何获取上述所有数据？提前致谢。

要抓取的数据集:

<div id="Category"><span>lists</span><ul>
 <li><a id="FR123456" onclick="setFood(false);setSeasonFruitID('123456');getit('call.php?category=fruits&amp;fruitid=123456&amp;',detailFruit,false);">mango season</a><img src="http://imagehosting.com/images/fru_123456.png">
 </li>



        <li><a id="FR123457" onclick="setFood(false);setSeasonFruitID('123457');getit('call.php?category=fruits&amp;fruitid=123457&amp;',detailFruit,false);">cherry season</a><img src="http://imagehosting.com/images/fru_123457.png">
                    </li>
 </ul></div>

python :

url ='http://somewebsite.com/data.php';

req = urllib2.Request(url)
req.add_header('User-Agent', 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0')
response = urllib2.urlopen(req)
responseData = response.read()

Answer 1

看一下 python 附带的 HTMLParser。我很快就会发布一些代码。 对于您的个人用例，这里有一些示例代码。这是一个漂亮的转储刮刀，但我认为你应该明白要点:

from HTMLParser import HTMLParser

# create a subclass and override the handler methods
class MyHTMLParser(HTMLParser):

    srces = []
    datas = []
    onclicks = []

    def handle_starttag(self, tag, attrs):
        for attr in attrs:
            if attr[0] == 'src':
                 self.srces.append(attr[1])
            if  attr[0] == 'onclick':
                  self.onclicks.append(attr[1].split("getit('")[1].split("',")[0])


    def handle_data(self, data):
        if data != " " and not data.startswith('\n'):
           self.datas.append(data)

parser = MyHTMLParser()
parser.feed(""" <li><a id="FR123456" onclick="setFood(false);setSeasonFruitID('123456');getit('call.php?category=fruits&amp;fruitid=123456&amp;',detailFruit,false);">mango season</a><img src="http://imagehosting.com/images/fru_123456.png">
                </li>

    <li><a id="FR123457" onclick="setFood(false);setSeasonFruitID('123457');getit('call.php?category=fruits&amp;fruitid=123457&amp;',detailFruit,false);">cherry season</a><img src="http://imagehosting.com/images/fru_123457.png">
                </li>"""
)


#To fit your output
for i in range(0,len(parser.srces)):
    print "1: " + parser.onclicks[i] + "\n" + "2: " + parser.datas[i] + "\n" + "3: " + parser.srces[i] + "\n"


#  Output
# 1: call.php?category=fruits&fruitid=123456&
# 2: mango season
# 3: http://imagehosting.com/images/fru_123456.png

# 1: call.php?category=fruits&fruitid=123457&
# 2: cherry season
# 3: http://imagehosting.com/images/fru_123457.png

编辑

回答您的后续问题:

base_url ='http://somewebsite.com/'
for i in range(0,len(parser.onclicks)):
    url = base_url + parser.onclicks[i][:-1] # use onclick without the last character (&)
    req = urllib2.Request(url)
    req.add_header('User-Agent', 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0')
    response = urllib2.urlopen(req)
    responseData = response.read()