我有一个包含很多关键字的数组:
array = ['table', 'chair', 'pen']
我想使用 python icrawler 从 Google 图片搜索中抓取 array 中每个项目的 5 张图片
这里是初始化:
from icrawler.builtin import GoogleImageCrawler
google_crawler = GoogleImageCrawler(
parser_threads=2,
downloader_threads=4,
storage={ 'root_dir': 'images' }
)
我使用循环来抓取每个项目:
for item in array:
google_crawler.crawl(
keyword=item,
offset=0,
max_num=5,
min_size=(500, 500)
)
但是，我收到错误日志:
File "crawler.py", line 20, in <module>
min_size=(500, 500)
File "/home/user/opt/miniconda3/envs/pak/lib/python3.6/site-packages/icrawler/builtin/google.py", line 83, in crawl
feeder_kwargs=feeder_kwargs, downloader_kwargs=downloader_kwargs)
File "/home/user/opt/miniconda3/envs/pak/lib/python3.6/site-packages/icrawler/crawler.py", line 166, in crawl
self.feeder.start(**feeder_kwargs)
File "/home/user/opt/miniconda3/envs/pak/lib/python3.6/site-packages/icrawler/utils/thread_pool.py", line 66, in start
worker.start()
File "/home/user/opt/miniconda3/envs/pak/lib/python3.6/threading.py", line 842, in start
raise RuntimeError("threads can only be started once")
RuntimeError: threads can only be started once
看来我不能多次使用google_crawler.crawl。我该如何解决这个问题？
最佳答案
在最新版本中，你可以这样使用。
from icrawler.builtin import GoogleImageCrawler
google_crawler = GoogleImageCrawler(
parser_threads=2,
downloader_threads=4,
storage={'root_dir': 'images'}
)
for keyword in ['cat', 'dog']:
google_crawler.crawl(
keyword=keyword, max_num=5, min_size=(500, 500), file_idx_offset='auto')
# set `file_idx_offset` to 'auto' will prevent naming the 5 images
# of dog from 000001.jpg to 000005.jpg, but naming it from 000006.jpg.
或者，如果您想将这些图像下载到不同的文件夹，您只需创建两个 GoogleImageCrawler 实例即可。
from icrawler.builtin import GoogleImageCrawler
for keyword in ['cat', 'dog']:
google_crawler = GoogleImageCrawler(
parser_threads=2,
downloader_threads=4,
storage={'root_dir': 'images/{}'.format(keword)}
)
google_crawler.crawl(
keyword=keyword, max_num=5, min_size=(500, 500))
关于python - 如何使用python icrawler爬取多个关键词，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/44292193/