我在使用 python 的多处理优雅地处理键盘中断时遇到问题
(是的,我知道 Ctr-C 不应该保证正常关闭——但让我们把这个讨论留到另一个线程)
考虑以下代码,我是 multiprocessing.Manager#list()
的用户,这是一个 ListProxy,据我所知,它处理对列表的多进程访问。
当我按 Ctr-C 退出时 -- 我在尝试访问 ListProxy 时收到 socket.error: [Errno 2] No such file or directory
我希望共享列表不会在 Ctr-C 时被破坏。这可能吗?!
注意:我想在不使用池和队列的情况下解决这个问题。
from multiprocessing import Process, Manager
from time import sleep
def f(process_number, shared_array):
try:
print "starting thread: ", process_number
shared_array.append(process_number)
sleep(3)
shared_array.append(process_number)
except KeyboardInterrupt:
print "Keyboard interrupt in process: ", process_number
finally:
print "cleaning up thread", process_number
if __name__ == '__main__':
processes = []
manager = Manager()
shared_array = manager.list()
for i in xrange(4):
p = Process(target=f, args=(i, shared_array))
p.start()
processes.append(p)
try:
for process in processes:
process.join()
except KeyboardInterrupt:
print "Keyboard interrupt in main"
for item in shared_array:
# raises "socket.error: [Errno 2] No such file or directory"
print item
如果您运行它然后按 Ctr-C,我们会得到以下信息:
starting thread: 0
starting thread: 1
starting thread: 3
starting thread: 2
^CKeyboard interupt in process: 3
Keyboard interupt in process: 0
cleaning up thread 3
cleaning up thread 0
Keyboard interupt in process: 1
Keyboard interupt in process: 2
cleaning up thread 1
cleaning up thread 2
Keyboard interupt in main
Traceback (most recent call last):
File "multi.py", line 33, in <module>
for item in shared_array:
File "<string>", line 2, in __getitem__
File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/managers.py", line 755, in _callmethod
self._connect()
File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/managers.py", line 742, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/connection.py", line 169, in Client
c = SocketClient(address)
File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/connection.py", line 293, in SocketClient
s.connect(address)
File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
socket.error: [Errno 2] No such file or directory
(这是另一种使用具有类似效果的 multiprocessing.Lock
的方法 ... gist )
类似问题:
最佳答案
multiprocessing.Manager() 启动一个子进程,负责处理您的共享列表代理。
运行时的 netstat 输出:
unix 2 [ ACC ] STREAM LISTENING 3921657 8457/python /tmp/pymp-B9dcij/listener-X423Ml
这个由 multiprocessing.Manager() 创建的子进程正在捕获您的 SIGINT 并退出导致与它相关的任何内容被取消引用因此您的“没有这样的文件”错误(我还遇到了其他几个错误,具体取决于我何时决定发送 SIGINT ).
要解决这个问题,您可以直接声明一个 SyncManager 对象(而不是让 Manager() 为您做)。这将要求您使用 start() 方法来实际启动子进程。 start() 方法将初始化函数作为其第一个参数(您可以在此处为管理器覆盖 SIGINT)。
下面的代码,试一试:
from multiprocessing import Process, Manager
from multiprocessing.managers import BaseManager, SyncManager
from time import sleep
import signal
#handle SIGINT from SyncManager object
def mgr_sig_handler(signal, frame):
print 'not closing the mgr'
#initilizer for SyncManager
def mgr_init():
signal.signal(signal.SIGINT, mgr_sig_handler)
#signal.signal(signal.SIGINT, signal.SIG_IGN) # <- OR do this to just ignore the signal
print 'initialized mananger'
def f(process_number, shared_array):
try:
print "starting thread: ", process_number
shared_array.append(process_number)
sleep(3)
shared_array.append(process_number)
except KeyboardInterrupt:
print "Keyboard interrupt in process: ", process_number
finally:
print "cleaning up thread", process_number
if __name__ == '__main__':
processes = []
#using syncmanager directly instead of letting Manager() do it for me
manager = SyncManager()
manager.start(mgr_init) #fire up the child manager process
shared_array = manager.list()
for i in xrange(4):
p = Process(target=f, args=(i, shared_array))
p.start()
processes.append(p)
try:
for process in processes:
process.join()
except KeyboardInterrupt:
print "Keyboard interrupt in main"
for item in shared_array:
print item
关于python - 键盘中断与 python 的多处理,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/21104997/