Runtime error loading numpy

[nkr0]

I've this c++ code. It has import numpy in a function which is called twice.

#include <iostream>
#include <pybind11/embed.h>

namespace py = pybind11;

void test() {
  py::scoped_interpreter guard;
  try {
    py::exec(R"(
      import numpy
      print("module loaded")
    )");
  } catch (py::error_already_set &e) {
    std::cerr << e.what() << "\n";
  }
}

int main() {
  // py::scoped_interpreter guard; // If the `guard` is in the `main` scope, there is no issue.
  test();
  test(); // this call fails to import numpy
  return 0;
}

All the python parts happen inside a function with a scoped_interpreter. I call the function twice from main. The first time, everything works. The second time it fails. And this is only if numpy is imported. Output error is different in different version of python and numpy. But I think it is related to numpy's CPU dispatcher.

module loaded
RuntimeError: CPU dispatcher tracer already initlized

At:
  /home/nkr/.local/lib/python3.11/site-packages/numpy/_core/overrides.py(8): <module>
  <frozen importlib._bootstrap>(241): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(940): exec_module
  <frozen importlib._bootstrap>(705): _load_unlocked
  <frozen importlib._bootstrap>(1150): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1176): _find_and_load
  <frozen importlib._bootstrap>(241): _call_with_frames_removed
  <frozen importlib._bootstrap>(1233): _handle_fromlist
  /home/nkr/.local/lib/python3.11/site-packages/numpy/_core/multiarray.py(10): <module>
  <frozen importlib._bootstrap>(241): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(940): exec_module
  <frozen importlib._bootstrap>(705): _load_unlocked
  <frozen importlib._bootstrap>(1150): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1176): _find_and_load
  <frozen importlib._bootstrap>(241): _call_with_frames_removed
  <frozen importlib._bootstrap>(1233): _handle_fromlist
  /home/nkr/.local/lib/python3.11/site-packages/numpy/_core/__init__.py(51): <module>
  <frozen importlib._bootstrap>(241): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(940): exec_module
  <frozen importlib._bootstrap>(705): _load_unlocked
  <frozen importlib._bootstrap>(1150): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1176): _find_and_load
  <frozen importlib._bootstrap>(241): _call_with_frames_removed
  <frozen importlib._bootstrap>(1126): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1176): _find_and_load
  /home/nkr/.local/lib/python3.11/site-packages/numpy/__config__.py(4): <module>
  <frozen importlib._bootstrap>(241): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(940): exec_module
  <frozen importlib._bootstrap>(705): _load_unlocked
  <frozen importlib._bootstrap>(1150): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1176): _find_and_load
  /home/nkr/.local/lib/python3.11/site-packages/numpy/__init__.py(115): <module>
  <frozen importlib._bootstrap>(241): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(940): exec_module
  <frozen importlib._bootstrap>(705): _load_unlocked
  <frozen importlib._bootstrap>(1149): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1176): _find_and_load
  <string>(3): <module>

[wenshitaiyi]

可以看一下这篇文章：https://stackoverflow.com/questions/48005042/deadlock-when-qthread-tries-to-acquire-python-gil-via-pygilstate-ensure

[nkr0]

This issue comes from Py_FinalizeEx. Basically, initialising and finalising python interpreter multiple times from a process can lead to undefined behaviour.

Some extensions may not work properly if their initialization routine is called more than once; this can happen if an application calls Py_Initialize() and Py_FinalizeEx() more than once.

My solution at this point is to use a single scoped_interpreter and use PyDict_Clear to clean variables from it. Not at all thread safe.

#include <pybind11/embed.h>

namespace py = pybind11;

void test(const char *code) {
  // Clear references to modules and variables from `__main__`
  PyDict_Clear(PyModule_GetDict(PyImport_AddModule("__main__")));
  py::exec(R"(
import numpy
print("module loaded")
    )");
  py::exec(code);
}

int main() {
  py::scoped_interpreter guard; 
  test("x=1; print('First time', x)");
  // I want this to fail with `NameError: name 'x' is not defined`
  test("print('Second time', x)");
  return 0;
}