python - boost.python C++ 多线程

标签 python c++ multithreading boost-python gil

我正在编写一个包含 C++ 模块的 Python 程序(.so,使用 boost.python)。
我正在启动几个运行 C++ 函数的 python 线程。

这是 C++ 代码的样子:

#include <boost/python.hpp>
using namespace boost;
void f(){
    // long calculation

    // call python function

    // long calculation
}

BOOST_PYTHON_MODULE(test)
{
    python::def("f", &f);
}

和 python 代码:

from test import f
t1 = threading.Thread(target=f)
t1.setDaemon(True)
t1.start()
print "Still running!"

我遇到一个问题:“仍在运行!”消息没有显示,我发现 C++ 线程持有 GIL。

在我从 python 代码运行 c++ 代码的情况下,处理 GIL 的最佳方法是什么?

谢谢! 加尔

最佳答案

我经常发现使用 RAII-style类来管理 Global Interpreter Lock (GIL) 提供了一个优雅的异常安全解决方案。

例如,对于以下 with_gil 类,当创建 with_gil 对象时,调用线程将获取 GIL。当 with_gil 对象被销毁时,它会恢复 GIL 状态。

/// @brief Guard that will acquire the GIL upon construction, and
///        restore its state upon destruction.
class with_gil
{
public:
  with_gil()  { state_ = PyGILState_Ensure(); }
  ~with_gil() { PyGILState_Release(state_);   }

  with_gil(const with_gil&)            = delete;
  with_gil& operator=(const with_gil&) = delete;
private:
  PyGILState_STATE state_;
};

而互补的 without_gil 类做相反的事情:

/// @brief Guard that will unlock the GIL upon construction, and
///        restore its staet upon destruction.
class without_gil
{
public:
  without_gil()  { state_ = PyEval_SaveThread(); }
  ~without_gil() { PyEval_RestoreThread(state_); }

  without_gil(const without_gil&)            = delete;
  without_gil& operator=(const without_gil&) = delete;
private:
  PyThreadState* state_;
};

它们在函数中的用法如下:

void f()
{
  without_gil no_gil;       // release gil
  // long calculation
  ...

  {
    with_gil gil;           // acquire gil
    // call python function
    ...
  }                         // restore gil (release)

  // long calculation
  ...
}                           // restore gil (acquire)

还可以使用更高级别的便利类来提供类似 std::lock_guard 的体验。 GIL 获取和释放、保存和恢复语义与普通互斥体略有不同。因此,gil_guard 接口(interface)是不同的:

  • gil_guard.acquire() 将获取 GIL
  • gil_guard.release() 将释放 GIL
  • gil_guard_restore() 会恢复之前的状态
/// @brief Guard that provides higher-level GIL controls.
class gil_guard
{
public:
  struct no_acquire_t {} // tag type used for gil acquire strategy
  static no_acquire;

  gil_guard()             { acquire(); }
  gil_guard(no_acquire_t) { release(); }
  ~gil_guard()            { while (!stack_.empty()) { restore(); } }

  void acquire()          { stack_.emplace(new with_gil); }
  void release()          { stack_.emplace(new without_gil); }
  void restore()          { stack_.pop(); }

  static bool owns_gil()
  {
    // For Python 3.4+, one can use `PyGILState_Check()`.
    return _PyThreadState_Current == PyGILState_GetThisThreadState();
  }

  gil_guard(const gil_guard&)            = delete;
  gil_guard& operator=(const gil_guard&) = delete;

private:
  // Use std::shared_ptr<void> for type erasure.
  std::stack<std::shared_ptr<void>> stack_;
};

它的用法是:

void f()
{
  gil_guard gil(gil_guard::no_acquire); // release gil
  // long calculation
  ...

  gil.acquire();                        // acquire gil
  // call python function
  ...
  gil.restore();                        // restore gil (release)

  // long calculation
  ...
}                                       // restore gil (acquire)

这是一个完整的例子demonstrating使用这些辅助类的 GIL 管理:

#include <cassert>
#include <iostream> // std::cout, std::endl
#include <memory>   // std::shared_ptr
#include <thread>   // std::this_thread
#include <stack>    // std::stack
#include <boost/python.hpp>

/// @brief Guard that will acquire the GIL upon construction, and
///        restore its state upon destruction.
class with_gil
{
public:
  with_gil()  { state_ = PyGILState_Ensure(); }
  ~with_gil() { PyGILState_Release(state_);   }

  with_gil(const with_gil&)            = delete;
  with_gil& operator=(const with_gil&) = delete;
private:
  PyGILState_STATE state_;
};

/// @brief Guard that will unlock the GIL upon construction, and
///        restore its staet upon destruction.
class without_gil
{
public:
  without_gil()  { state_ = PyEval_SaveThread(); }
  ~without_gil() { PyEval_RestoreThread(state_); }

  without_gil(const without_gil&)            = delete;
  without_gil& operator=(const without_gil&) = delete;
private:
  PyThreadState* state_;
};

/// @brief Guard that provides higher-level GIL controls.
class gil_guard
{
public:
  struct no_acquire_t {} // tag type used for gil acquire strategy
  static no_acquire;

  gil_guard()             { acquire(); }
  gil_guard(no_acquire_t) { release(); }
  ~gil_guard()            { while (!stack_.empty()) { restore(); } }

  void acquire()          { stack_.emplace(new with_gil); }
  void release()          { stack_.emplace(new without_gil); }
  void restore()          { stack_.pop(); }

  static bool owns_gil()
  {
    // For Python 3.4+, one can use `PyGILState_Check()`.
    return _PyThreadState_Current == PyGILState_GetThisThreadState();
  }

  gil_guard(const gil_guard&)            = delete;
  gil_guard& operator=(const gil_guard&) = delete;

private:
  // Use std::shared_ptr<void> for type erasure.
  std::stack<std::shared_ptr<void>> stack_;
};

void f()
{
  std::cout << "in f()" << std::endl;

  // long calculation
  gil_guard gil(gil_guard::no_acquire);
  assert(!gil.owns_gil());
  std::this_thread::sleep_for(std::chrono::milliseconds(500));
  std::cout << "calculating without gil..." << std::endl;

  // call python function
  gil.acquire();
  assert(gil.owns_gil());
  namespace python = boost::python;
  python::object print =
  python::import("__main__").attr("__builtins__").attr("print");
    print(python::str("calling a python function"));
  gil.restore();

  // long calculation
  assert(!gil.owns_gil());
  std::cout << "calculating without gil..." << std::endl;
}

BOOST_PYTHON_MODULE(example)
{
  // Force the GIL to be created and initialized.  The current caller will
  // own the GIL.
  PyEval_InitThreads();

  namespace python = boost::python;
  python::def("f", +[] {
    // For exposition, assert caller owns GIL before and after
    // invoking function `f()`.
    assert(gil_guard::owns_gil());
    f();
    assert(gil_guard::owns_gil());
  });
}

交互使用:

>>> import threading
>>> import example
>>> t1 = threading.Thread(target=example.f)
>>> t1.start(); print "Still running"
in f()
Still running
calculating without gil...
calling a python function
calculating without gil...
>>> t1.join()

关于python - boost.python C++ 多线程,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/41133001/

相关文章:

来自索引数组的python numpy数组

c++ - 循环中变量的范围

python - 为什么 super().__init__ 在子类化 threading.Thread 时不起作用?

c++ - Qt Blocking Master 示例中 Mutex 的使用

Python API 兼容性检查器

python - 找出游戏板上两个正方形之间的最短距离

python - selenium.common.exceptions.InvalidSelectorException : Message: invalid selector: An invalid or illegal selector was specified

c++ - opencv:对 "Mat"的调用不明确

c++ - 使用列名 vector 生成 sql 语句

尝试将字符串编码/解码为 UTF-8 后引发 Java 服务器线程空指针异常错误