Python 多处理队列 NotImplementedError macOS

Question

系统信息

python 3.8.7

操作系统 11.1(大苏尔)

通过 brew install python@3.8 安装的 Python

要在 Big Sur 和最有可能的旧版本上重现:

import multiprocessing as mp


if __name__ == '__main__':
    exp_queue = mp.Queue()
    print(exp_queue.qsize())

结果:

  File "/Users/username/Library/Application Support/JetBrains/PyCharm2020.3/scratches/scratch.py", line 5, in <module>
    print(exp_queue.qsize())
  File "/usr/local/Cellar/python@3.8/3.8.7/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/queues.py", line 120, in qsize
    return self._maxsize - self._sem._semlock._get_value()
NotImplementedError

看起来是谁在 multiprocessing/queues.py line 120 中写的知道这个问题，但我在某处找不到解决方案:

def qsize(self):
    # Raises NotImplementedError on Mac OSX because of broken sem_getvalue()
    return self._maxsize - self._sem._semlock._get_value()

Answer 1

正如 Víctor Terrón 在 GitHub 讨论中所建议的那样，您可以使用他的实现:
https://github.com/vterron/lemon/blob/d60576bec2ad5d1d5043bcb3111dff1fcb58a8d6/methods.py#L536-L573
根据文档:

A portable implementation of multiprocessing.Queue. Because of multithreading / multiprocessing semantics, Queue.qsize() may raise the NotImplementedError exception on Unix platforms like Mac OS X where sem_getvalue() is not implemented. This subclass addresses this problem by using a synchronized shared counter (initialized to zero) and increasing / decreasing its value every time the put() and get() methods are called, respectively. This not only prevents NotImplementedError from being raised, but also allows us to implement a reliable version of both qsize() and empty().