我正在使用 Python Requests Module对网站进行数据挖掘。作为数据挖掘的一部分,我必须通过 HTTP POST 表单并通过检查生成的 URL 来检查它是否成功。我的问题是,在 POST 之后,是否可以请求服务器不发送整个页面?我只需要检查 URL,但我的程序下载了整个页面并消耗了不必要的带宽。代码很简单
import requests
r = requests.post(URL, payload)
if 'keyword' in r.url:
success
fail
最佳答案
一个简单的解决方案,如果它对您来说是可行的。就是走低级。使用套接字库。 例如,您需要发送一个正文中包含一些数据的 POST。我在一个站点的爬虫中使用了它。
import socket
from urllib import quote # POST body is escaped. use quote
req_header = "POST /{0} HTTP/1.1\r\nHost: www.yourtarget.com\r\nUser-Agent: For the lulz..\r\nContent-Type: application/x-www-form-urlencoded; charset=UTF-8\r\nContent-Length: {1}"
req_body = quote("data1=yourtestdata&data2=foo&data3=bar=")
req_url = "test.php"
header = req_header.format(req_url,str(len(req_body))) #plug in req_url as {0}
#and length of req_body as Content-length
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM) #create a socket
s.connect(("www.yourtarget.com",80)) #connect it
s.send(header+"\r\n\r\n"+body+"\r\n\r\n") # send header+ two times CR_LF + body + 2 times CR_LF to complete the request
page = ""
while True:
buf = s.recv(1024) #receive first 1024 bytes(in UTF-8 chars), this should be enought to receive the header in one try
if not buf:
break
if "\r\n\r\n" in page: # if we received the whole header(ending with 2x CRLF) break
break
page+=buf
s.close() # close the socket here. which should close the TCP connection even if data is still flowing in
# this should leave you with a header where you should find a 302 redirected and then your target URL in "Location:" header statement.
关于Python 请求 - 是否可以在 HTTP POST 后收到部分响应?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/13170258/