我有一个数据分析脚本,它接受一个参数来指定要执行的分析部分。我想一次运行最多“n”个脚本实例,其中“n”是机器上的核心数。复杂的是分析的部分多于核心,所以我想一次最多运行“n”个进程,其中一个完成,启动另一个。有没有人在使用 subprocess 模块之前做过这样的事情?
最佳答案
我认为 multiprocessing 模块将帮助您实现您的需求。 查看示例技术。
import multiprocessing
def do_calculation(data):
"""
@note: you can define your calculation code
"""
return data * 2
def start_process():
print 'Starting', multiprocessing.current_process().name
if __name__ == '__main__':
analsys_jobs = list(range(10)) # could be your analysis work
print 'analsys_jobs :', analsys_jobs
pool_size = multiprocessing.cpu_count() * 2
pool = multiprocessing.Pool(processes=pool_size,
initializer=start_process,
maxtasksperchild=2, )
#maxtasksperchild = tells the pool to restart a worker process \
# after it has finished a few tasks. This can be used to avoid \
# having long-running workers consume ever more system resources
pool_outputs = pool.map(do_calculation, analsys_jobs)
#The result of the map() method is functionally equivalent to the \
# built-in map(), except that individual tasks run in parallel. \
# Since the pool is processing its inputs in parallel, close() and join()\
# can be used to synchronize the main process with the \
# task processes to ensure proper cleanup.
pool.close() # no more tasks
pool.join() # wrap up current tasks
print 'Pool :', pool_outputs
你可以找到好的multiprocessing techniques here开始
关于python - 使用 python 进行子进程管理,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/12766026/