This是我要抓取的页面
页面上的数据来自这个URL
这是我的爬虫代码。我至少检查了 5 次标题和表单数据。我认为他们是正确的。问题是 scrapy 向 start_url 发送了一个 GET 请求，即使我重写了 parse 方法的默认行为。
class MySpider(CrawlSpider):
name = 'myspider'
start_urls = [
'https://277kmabdt6-dsn.algolia.net/1/indexes/*/queries?x-algolia-agent=Algolia%20for%20vanilla%20JavaScript%20(lite)%203.27.1%3BJS%20Helper%202.26.0%3Bvue-instantsearch%201.7.0&x-algolia-application-id=277KMABDT6&x-algolia-api-key=bf8b92303c2418c9aed3c2f29f6cbdab',
]
formdata = {
'requests': [{'indexName': 'listings',
'params': 'query=&hitsPerPage=24&page=0&highlightPreTag=__ais-highlight__&highlightPostTag=__%2Fais-highlight__&filters=announce_type%3Aproperty-announces%20AND%20language_code%3Apt%20AND%20listing_id%3A%205&facets=%5B%22announce_type%22%5D&tagFilters='}]
}
headers = {
'accept': 'application/json',
'content-type': 'application/x-www-form-urlencoded',
'Origin': 'https://www.flat.com.br',
'Referer': 'https://www.flat.com.br/search?query=',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36',
}
def parse(self, response):
for url in self.start_urls:
yield scrapy.FormRequest(
url=url,
method='POST',
headers=self.headers,
formdata=self.formdata,
callback=self.parse_page,
)
def parse_page(self, response):
print json.loads(response.text)
这是我在运行蜘蛛时收到的消息。
我的问题是；为什么 scrapy 向 url 发送 GET 请求，我是不是遗漏了什么？可能是我的请求失败的任何其他原因？
2019-07-01 11:45:58 [scrapy] DEBUG: Crawled (400) <GET https://277kmabdt6-dsn.algolia.net/1/indexes/*/queries?x-algolia-agent=Algolia%20for%20vanilla%20JavaScript%20(lite)%203.27.1%3BJS%20Helper%202.26.0%3Bvue-instantsearch%201.7.0&x-algolia-application-id=277KMABDT6&x-algolia-api-key=bf8b92303c2418c9aed3c2f29f6cbdab> (referer: None)
2019-07-01 11:45:58 [scrapy] DEBUG: Ignoring response <400 https://277kmabdt6-dsn.algolia.net/1/indexes/*/queries?x-algolia-agent=Algolia%20for%20vanilla%20JavaScript%20(lite)%203.27.1%3BJS%20Helper%202.26.0%3Bvue-instantsearch%201.7.0&x-algolia-application-id=277KMABDT6&x-algolia-api-key=bf8b92303c2418c9aed3c2f29f6cbdab>: HTTP status code is not handled or not allowed
最佳答案
您需要将parse 方法重命名为start_requests，因为默认情况下Scrapy 将为self.start_urls< 中的每个URL :GET/
def start_requests(self):
for url in self.start_urls:
yield scrapy.FormRequest(
url=url,
method='POST',
headers=self.headers,
formdata=self.formdata,
callback=self.parse_page,
)
关于python - Scrapy - FormRequest 在方法为 POST 时发送 GET 请求，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/56832803/