python - 身份验证后使用 Python 请求下载 CSV 文件

Question

我正在尝试以编程方式从网络下载“.csv”文件。我收到包含以下代码的 200 响应。

import requests
url = 'http://eoddata.com/data/filedownload.aspx?e=NYSE&sd=20180722&ed=20180722&d=5&k=xh2mnktdc4&o=d&ea=1&p=0'
user, password = '..', '..'
resp = requests.get(url, auth=(user, password))

在浏览器中提交此 URL 时，会自动下载“.csv”文件。我正在尝试让我的应用程序在本地存储文件。我不知道文件的结束位置，因此无法保存它。

我添加了 HTTP Live headers 扩展，它给了我这个:

GET /data/filedownload.aspx?e=AMEX&sd=20180719&ed=20180719&d=5&k=n4vic5se64&o=d&ea=1&p=0 HTTP/1.1
Host: eoddata.com
Accept: text/css,*/*;q=0.1
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Cookie: ASP.NET_SessionId=drbcwljllaf3xdwdlvqzfvl1;EODDataAdmin=01438A3826CBEEC6085BF7B27AEB1BE6AF337FFBF3E3E33AA1D882D39FB92555E8370DD45631B3F8CAD5289C5C46DB28C99B44DC8253A78D659EB165ADC55A1D6E4C20FA81051908E6BE82143EA5398662779546BE4C31C6F32833C5558F23C9A30D0CA16A703CB939FC6D77A0FB0CB017DDEEA832A1A0CC228B61777295C58CA52AF78B472B9D2FC19E78770DEBD346E9B05F02C65F96EE22C3E55B083FC4A1311FA12B223CC2899D6166E1284AD95E
DNT: 1
Referer: http://eoddata.com/download.aspx
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36

HTTP/1.1 200 OK
Cache-Control: private
content-disposition: attachment;filename="AMEX_20180719.csv"
Content-Length: 93840
Content-Type: text/plain
Date: Wed, 25 Jul 2018 13:54:31 GMT
Server: Microsoft-IIS/7.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET

非常感谢您在保存预期文件方面的任何帮助。

编辑: 我想在 header 响应中指出这一行: content-disposition:attachment;filename="AMEX_20180719.csv" 这有什么区别吗？

Answer 1

试试这个:

import requests, urllib
from tqdm import tqdm

def download_file(url):

    filename = "FILENAME.csv"
    print("Downloading %s ..." % filename)

    with open(filename, "wb") as handle:
        for data in tqdm(response.iter_content()):
            handle.write(data)

    print("Download complete for %s!" % filename)

username = "USERNAME"
password = "PASSWORD"

data_credentials = {'user_id': username, 'password': password}
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36'}

login_url = "https://mywebsite.com/login/"

requests_session = requests.Session() # create a requests Session

response = requests_session.post(login_url, headers=headers, data=data_credentials) # log in to the requests Session so that you can reuse it

download_url = "http://www.downloadmyfile.com/something.csv"

response = requests_session.get(download_url, timeout=(15,15))

download_file(response.url) # run function to download file