我正在编写一个包含 20,000 个任务的脚本,每个任务进行一个子进程调用和一个或两个 TCP 调用。为了使这不会花费一整天,我使用了 Python 的新 asyncio
。
但是,我担心脚本运行时 Python 会打印出这些错误:
Exception ignored when trying to write to the signal wakeup fd:
BlockingIOError: [Errno 11] Resource temporarily unavailable
它将打印一堆,但不会引发任何异常。我确实得到了关于太多打开的文件
的OSError
并且之前与服务器断开了连接,但是我使用信号量一次只允许100个连接到每个服务器并且只有700个连接总数。
由于 Python 没有引发任何异常,我无法捕获错误。但它似乎不会影响脚本。
我应该担心这些错误吗?如果是这样,我需要做什么来摆脱它们?如果没有,我该如何摆脱它们,使它们不在我的程序输出中?
此外,如果这些错误很严重,为什么 Python 会忽略它们而不是引发异常?
最佳答案
看起来限制因素是运行大量短命的 subprocess
es。来自Python bug tracker :
"Exception ignored when trying to write to the signal wakeup fd" message comes from the signal handler in Modules/signalmodule.c. The problem is that Python gets a lot of SIGCHLD signals (the test scripts creates +300 processes per second on my computer). The producer (signal handler writing the signal number into the "self" pipe) is faster than the consumer (BaseSelectorEventLoop._read_from_self callback).
With the patch, I start getting messages with 140 concurrent processes, which is much better :-) IMO more than 100 concurrent processes is crazy, don't do that at home :-) I mean processes with a very short lifetime. The limit is the number of SIGCHLD per second, so the number of processes which end at the same second.
我更改了我的代码以限制多少 create_subprocess_exec
可以同时运行。当我低于 35 时,我不再看到错误,尽管我可能会将其设置为 20 以确保万无一失。您的里程可能会有所不同。
async def myTask(stuff, semaphore, loop):
with semaphore:
process = await asyncio.create_subprocess_exec('short_program', loop=loop)
def taskRunner(stuffs):
loop = asyncio.get_event_loop()
semaphore = asyncio.Semaphore(20) # limit how many can run at a time
tasks = [
asyncio.ensure_future(myTask(semaphore, loop))
for i in range(20000)
]
loop.run_until_complete(asyncio.gather(*tasks))
loop.close()
关于python - BlockingIOError 异常被忽略;我应该担心吗?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/52376095/