<分区>
我尝试抓取 example.com,但在抓取 100 个页面后,该网站被阻止了。
我该如何纠正?
AWS 是否有助于避免阻塞?
<分区>
我尝试抓取 example.com,但在抓取 100 个页面后,该网站被阻止了。
我该如何纠正?
AWS 是否有助于避免阻塞?
最佳答案
请参阅 scrapy faq page 上的说明:
Avoiding getting banned Some websites implement certain measures to prevent bots from crawling them, with varying degrees of sophistication. Getting around those measures can be difficult and tricky, and may sometimes require special infrastructure. Please consider contacting commercial support if in doubt.
Here are some tips to keep in mind when dealing with these kind of sites:
- rotate your user agent from a pool of well-known ones from browsers (google around to get a list of them)
- disable cookies (see COOKIES_ENABLED) as some sites may use cookies to spot bot behaviour
- use download delays (2 or higher). See DOWNLOAD_DELAY setting. if possible, use Google cache to fetch pages, instead of hitting the sites directly
- use a pool of rotating IPs. For example, the free Tor project or paid services like ProxyMesh
- use a highly distributed downloader that circumvents bans internally, so you can just focus on parsing clean pages. One example of such downloaders is Crawlera
If you are still unable to prevent your bot getting banned, consider contacting commercial support.
关于Python Scrapy - IP 网络掩码,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/21437718/