python-3.x - 在父进程中捕获子进程异常

Question

我正在创建多个单独运行爬网程序的进程。我想确保如果爬虫进程中有一些异常，我能够在父进程中捕获它。 这是流程创建代码:

try:
    caching_process = Process(target=run_crawler_process, args=(Config.CRAWLER_NAME, locations,
                                                               city_payloads_map, cycle_count))
    caching_process.start()
except Exception as processException:
    raise processException

Answer 1

您不能使用 Process 对象来做到这一点。 multiprocessing.Pool或concurrent.futures.ProcessPoolExecutor允许这样做。

pool = multiprocessing.Pool()
task = pool.apply_async(run_crawler_process, (Config.CRAWLER_NAME, locations,c ity_payloads_map, cycle_count))

try:
    task.get()
except Exception as error:  
    print("Error while processing task: %s" % error)