python - 使用 Twill 从登录页面 Python 中抓取 .txt

Question

我正在使用 Twill 检索包含所需 .txt 数据的页面，以便将它们存储为 Excel 文件。数据受密码保护，因此我从 /user/login 页面登录。

我的代码遇到了问题，它试图从登录屏幕访问文本页面并遇到了 HTML 的砖墙而不是 .txt 本身。

当我运行登录时:

path = "https://naturalgasintel.com/ext/resources/Data-Feed/Daily-GPI/"
end = "td.txt"

go("http://www.naturalgasintel.com/user/login")
showforms()
fv("2", "user[email]", user_email)
fv("2", "user[password]", user_password)
fv("2", "commit", "Login")

datafilelocation = path + year + "/" + month + "/" + date + end
go(datafilelocation)

当我的代码到达 go(datafilelocation) 我得到这个:

==> at https://www.naturalgasintel.com/user/login?referer=%2Fext%2Fresources%2FData-Feed%2FDaily-GPI%2F2018%2F12%2F20181221td.txt
Out[18]: u'https://www.naturalgasintel.com/user/login?referer=%2Fext%2Fresources%2FData-Feed%2FDaily-GPI%2F2018%2F12%2F20181221td.txt'

所以当我真的想访问页面时，它指向 referer 而不是实际的文本:

https://naturalgasintel.com/ext/resources/Data-Feed/Daily-GPI/2018/12/20181221td.txt

我使用 fv("2", "commit", "Login") 而不是 submit() 的原因是当我到达页面时它给出我这个:

显示形式()

Form name=quick-search (#1)
## ## __Name__________________ __Type___ __ID________ __Value__________________
1     q                        text      q            Search 


Form #2
## ## __Name__________________ __Type___ __ID________ __Value__________________
1     utf8                     hidden    (None)       ✓ 
2     authenticity_token       hidden    (None)       pnFnPGhMomX2Lyh7/U8iGOZKsiQnyicj7BWT ... 
3     referer                  hidden    (None)       https://www.naturalgasintel.com/ext/ ... 
4     popup                    hidden    (None)       false 
5     user[email]              text      user_email    
6     user[password]           password  user_pas ... 
7     user[remember_me]        hidden    (None)       0 
8     user[remember_me]        checkbox  user_rem ... None 
9     commit                   submit    (None)       Login 

然后在我 submit() 之后它告诉我:

Note: submit is using submit button: name="commit", value="Login"

解决这个问题的最佳方案是什么？

Answer 1

如果您愿意使用 Mechanize 而不是 Twill，请尝试以下方法:

import mechanize

username = ""
password = ""
login_post_url = "http://www.naturalgasintel.com/user/login"
internal_url = "https://naturalgasintel.com/ext/resources/Data-Feed/Daily-GPI/2018/12/20181221td.txt"

browser = mechanize.Browser()
browser.open(login_post_url)
browser.select_form(nr = 1)
browser.form['user[email]'] = username
browser.form['user[password]'] = password
browser.submit()

response = browser.open(internal_url)
print response.read()