python-3.x - 如何停止重定向并停留在页面上以抓取数据(python3、beautifulsoup、urllib)

Question

我正在尝试从此处的网站上抓取评论:http://movie.douban.com/subject/2303845/comments

而且显然它不断将自己重定向回主页。我想找到一种方法来停止重定向并停留在同一页面上以获取具有 div 的 html 部分，并将评论作为其类。

关键词:Python3、Beautifulsoup、urllib

我已经阅读了其他帖子，下面是我尝试过的:

#!/usr/bin/env python
# -*- coding: utf-8 -*- 
import sys
from bs4 import BeautifulSoup
from urllib.request import urlopen

sys.setrecursionlimit(30000)

try:
    import urllib.request as urllib2
    import urllib.request as urllib
except ImportError:
    import urllib2

class NoRedirectHandler(urllib2.HTTPRedirectHandler):
    def http_error_302(self, req, fp, code, msg, headers):
        infourl = urllib.addinfourl(fp, headers, req.get_full_url())
        infourl.status = code
        infourl.code = code
        return infourl
    http_error_300 = http_error_302
    http_error_301 = http_error_302
    http_error_303 = http_error_302
    http_error_307 = http_error_302

opener = urllib2.build_opener(NoRedirectHandler())
urllib2.install_opener(opener)
response = urllib2.urlopen('http://movie.douban.com/subject/2303845/comments')
soup = BeautifulSoup(response, "html.parser")
comments = soup.findAll('div', {'class': 'comment'})
print (comments)

if response.code in (300, 301, 302, 303, 307):
    print ('redirect')

出于某种原因，我认为它仍然没有获取数据，我怀疑它永远不会停止重定向到另一个页面。

Answer 1

将 cookie 添加到标题 类似的东西(只需检查实际浏览器对 bean 瓣的设置):

标题= { “用户代理”:random_user_agent(), "Cookie": 'bid="' + "".join(random.sample(string.ascii_letters + string.digits, 11)) + '"; gr_user_id=42bc6859-9086-4933-9f19-8f2fc412c37a;查看=“5312099_10355142_4742942_3223711_26642036_25850008_4727293_10580248_3726306”； _ga=GA1.2.1107723060.1458721162； ll="108288"; ps=y; ue="276502214@qq.com"; dbcl2="48221338:qY8I57pvNs8"; ck=xvAF； ap=1； _pk_ref.100001.4cf6=%5B%22%22%2C%22%22%2C1487574236%2C%22https%3A%2F%2Fwww.douban.com%2Faccounts%2Flogin%3Fredir%3Dhttps%253A%252F%252Fmovie.douban. com%252Fsubject%252F2973543%252F%22%5D； _pk_id.100001.4cf6=ccee4c5d3437a382.1482239227.55.1487574252.1487571075.; _pk_ses.100001.4cf6=*； __utma=30149280.1107723060.1458721162.1487570089.1487574236.345; __utmb=30149280.0.10.1487574236； __utmc=30149280； __utmz=30149280.1487518823.338.100.utmcsr=百度|utmccn=(有机)|utmcmd=有机； __utmv=30149280.4822； __utma=223695111.1107723060.1458721162.1487570089.1487574236.52; __utmb=223695111.0.10.1487574236； __utmc=223695111； __utmz=223695111.1487298209.35.8.utmcsr=douban.com|utmccn=(referral)|utmcmd=referral|utmcct=/accounts/login; push_noty_num=0; push_doumail_num=0; _vwo_uuid_v2=2D161B768A99F5F485C8FFD318CE8704|b630fad39ce1f3cbdb2e7e44cc9411ca' } req = urllib.request.Request( 网址， 数据=无， 标题 = 标题 )