我设法使用 scrapy+splash 连接到网站(感谢 this thread )。
我知道我已登录,因为我可以显示您登录后可用的一些元素。但是,当我尝试使用另一个 SplashRequest
访问另一个页面时,网站会要求再次登录。
看来 scrapy(或splash)并不能保持 session 处于事件状态。是否需要启用某些功能才能保持登录状态并保持 session 事件?
谢谢,
最佳答案
Splash从干净的状态开始每次渲染,所以如果你想保持 session ,你需要首先初始化cookie,并且让Scrapy知道渲染期间设置的cookie。请参阅Session Handling scrapy-splash README 中的部分。完整的示例如下所示(从自述文件中复制粘贴):
import scrapy
from scrapy_splash import SplashRequest
script = """
function main(splash)
splash:init_cookies(splash.args.cookies)
assert(splash:go{
splash.args.url,
headers=splash.args.headers,
http_method=splash.args.http_method,
body=splash.args.body,
})
assert(splash:wait(0.5))
local entries = splash:history()
local last_response = entries[#entries].response
return {
url = splash:url(),
headers = last_response.headers,
http_status = last_response.status,
cookies = splash:get_cookies(),
html = splash:html(),
}
end
"""
class MySpider(scrapy.Spider):
# ...
yield SplashRequest(url, self.parse_result,
endpoint='execute',
cache_args=['lua_source'],
args={'lua_source': script},
)
def parse_result(self, response):
# here response.body contains result HTML;
# response.headers are filled with headers from last
# web page loaded to Splash;
# cookies from all responses and from JavaScript are collected
# and put into Set-Cookie response header, so that Scrapy
# can remember them.
请注意, session 当前需要使用/execute 或/run 端点,其他端点没有帮助程序。
关于python - Scrapy Splash - 保持记录,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/45328914/