python - 如何通过 selenium 将成功登录的状态数据合并到请求 session 中？

Question

当然，我当前的需求可以通过selenium来完成，但它有缺点:速度慢，需要在计算机上打开真正的浏览器。

我的计划是这样的:使用selenium登录获取一些状态数据，然后使用这些数据通过请求建立 session ，这样我就可以频繁地进行其他操作更快地发布请求(例如向本网站提交评论或帖子)。

我的尝试是直接修改 session 的属性，但失败并出现错误 requests.cookies.CookieConflictError: There are multiple cookies with name, 'tx3_sid'。

我尝试的原因是，当纯粹使用请求时:

import requests

loginUrl = "http://tx3.netease.com/logging.php?action=login"
# postUrl = "http://tx3.netease.com/post.php?action=reply&fid=22&tid=3966375"

client = requests.session()
r = client.get(loginUrl)
r.encoding='gb18030'

print(client.cookies)

# Will get: <<class 'requests.cookies.RequestsCookieJar'>[<Cookie tx3_sid=kkfBk7 for .netease.com/>, <Cookie PHPSESSID=6vctcjdrg1mrig6h0kndmuurn4 for tx3.netease.com/>]>
# So I try to get tx3_sid and PHPSESSID via selenium and set the attributes in requests `client.cookies`

这是我的尝试:

import requests
import json
import re
from os.path import dirname, realpath
from time import sleep
from selenium import webdriver

USERNAME = "smaller9@163.com"
PASSWORD = "111111"
loginUrl = "http://tx3.netease.com/logging.php?action=login"
postUrl = "http://tx3.netease.com/post.php?action=reply&fid=22&tid=3966375"

# create a driver
driver = webdriver.Firefox()
# get the homepage
driver.get(loginUrl)

un_elm = driver.find_element_by_id("username")
pw_elm = driver.find_element_by_id("ori_password")
submit = driver.find_element_by_css_selector("[name=loginsubmit]")

un_elm.send_keys(USERNAME)
pw_elm.send_keys(PASSWORD)

# click submit
submit.click()

for d in driver.get_cookies():
    if d['name']=='tx3_sid':
        tx3_sid = d['value']
    elif d['name']=='PHPSESSID':
        phpsessid = d['value']
print(tx3_sid, phpsessid)
# above are all OK. varibles `tx3_sid` and `phpsessid` are set.

# -------------------below is requests part. ------------------------
client = requests.session()
client.get(loginUrl)
# cookies are set
print(client.cookies['tx3_sid'],client.cookies['PHPSESSID'])
# here is my try -- failed
client.cookies['tx3_sid'] = tx3_sid
client.cookies['PHPSESSID'] = phpsessid

# so the following cannot be tested yet.
r = client.get(postUrl)
r.encoding='gb18030'
# grab `formhash` which is needed for post dict.
formhashPat = re.compile('<input type="hidden" name="formhash" value="([\d\w]+?)" />')
formhash = formhashPat.search(r.text).groups()[0]
answerData={
    'message':'this_is_post_content', 
    'formhash':formhash,
    'subject':'',
    'usesig':1,
}
client.post(url=postUrl, data=answerData)

更新，终于找到修改cookie值的方法了:

# here is my try -- failed
rck = client.cookies._cookies
#print(rck, type(rck))

for domain in rck:
    for path in rck[domain]:
        for name, cookie in rck[domain][path].items():
            if name == 'tx3_sid':
                cookie.value = tx3_sid
            elif name == 'PHPSESSID':
                cookie.value = phpsessid

但是 session 不再支持进一步的请求，所以也许这整个方法是行不通的。

Answer 1

requests 提供的字典接口(interface)在使用 cookie 时受到限制。

除了名称之外，Cookie 还具有域和路径，当您使用 client.cookies[name] = value 设置 Cookie 时，这两个域和路径均保留为空。 现有 Cookie 具有与其关联的域和路径，如果您想替换这些 Cookie，您需要先删除它们或匹配它们的域和路径。

首先删除它们是最简单的:

del client.cookies['tx3_sid'], client.cookies['PHPSESSID']
client.cookies['tx3_sid'] = tx3_sid
client.cookies['PHPSESSID'] = phpsessid

这会创建带有空白路径和域的新 Cookie，这意味着它们将用于没有特定域和路径匹配的所有请求。

另一种方法是循环客户端，找到匹配的cookie对象并直接更改它们:

for cookie in client.cookies:
    if cookie.name == 'tx3_sid':
        cookie.value = tx3_sid
    elif cookie.name == 'PHPSESSID':
        cookie.value = phpsessid