python - Scrapy中间件设置

Question

我正在尝试使用 scrapy 访问公共(public)代理来获取一些数据。当我尝试运行代码时出现以下错误:

ImportError: Error loading object 'craiglist.middlewares.ProxyMiddleware': No module named middlewares

我使用以下代码创建了 middlewares.py 文件:

import base64
# Start your middleware class
class ProxyMiddleware(object):
    # overwrite process request
    def process_request(self, request, spider):
        # Set the location of the proxy
        request.meta['proxy'] = "http://124.200.36.150:8118"

设置.py:

DOWNLOADER_MIDDLEWARES = {
    'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 110,
    'craiglist.middlewares.ProxyMiddleware': 100,
}

欢迎任何帮助。

Answer 1

通过此设置，您需要将 middlewares.py 向上移动一级到 craiglist 包中。