我想从一个网站获取很多页面,比如
curl "http://farmsubsidy.org/DE/browse?page=[0000-3603]" -o "de.#1"
但在 python 中获取页面数据,而不是磁盘文件。
有人可以发布 pycurl
代码来执行此操作吗,
或快速 urllib2
(不是一次一个),
或者说“算了,curl 更快更健壮”?谢谢
最佳答案
所以你有 2 个问题,让我在一个例子中向你展示。请注意,pycurl 已经完成了多线程处理/而不是一次一个处理,而无需您的辛勤工作。
#! /usr/bin/env python
import sys, select, time
import pycurl,StringIO
c1 = pycurl.Curl()
c2 = pycurl.Curl()
c3 = pycurl.Curl()
c1.setopt(c1.URL, "http://www.python.org")
c2.setopt(c2.URL, "http://curl.haxx.se")
c3.setopt(c3.URL, "http://slashdot.org")
s1 = StringIO.StringIO()
s2 = StringIO.StringIO()
s3 = StringIO.StringIO()
c1.setopt(c1.WRITEFUNCTION, s1.write)
c2.setopt(c2.WRITEFUNCTION, s2.write)
c3.setopt(c3.WRITEFUNCTION, s3.write)
m = pycurl.CurlMulti()
m.add_handle(c1)
m.add_handle(c2)
m.add_handle(c3)
# Number of seconds to wait for a timeout to happen
SELECT_TIMEOUT = 1.0
# Stir the state machine into action
while 1:
ret, num_handles = m.perform()
if ret != pycurl.E_CALL_MULTI_PERFORM:
break
# Keep going until all the connections have terminated
while num_handles:
# The select method uses fdset internally to determine which file descriptors
# to check.
m.select(SELECT_TIMEOUT)
while 1:
ret, num_handles = m.perform()
if ret != pycurl.E_CALL_MULTI_PERFORM:
break
# Cleanup
m.remove_handle(c3)
m.remove_handle(c2)
m.remove_handle(c1)
m.close()
c1.close()
c2.close()
c3.close()
print "http://www.python.org is ",s1.getvalue()
print "http://curl.haxx.se is ",s2.getvalue()
print "http://slashdot.org is ",s3.getvalue()
最后,这些代码主要是基于pycurl网站上的例子=.=
也许您真的应该阅读文档。人们花大量时间在上面。
关于python - 使用 pycurl 获取很多页面?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/1959240/