How To Reload A Python3 C Extension Module?

November 16, 2024 Post a Comment

I wrote a C extension (mycext.c) for Python 3.2. The extension relies on constant data stored in a C header (myconst.h). The header file is generated by a Python script. In the sam

Solution 1:

~~You can reload modules in Python 3.x by using the imp.reload() function. (This function used to be a built-in in Python 2.x. Be sure to read the documentation -- there are a few caveats!)~~

Python's import mechanism will never dlclose() a shared library. Once loaded, the library will stay until the process terminates.

Your options (sorted by decreasing usefulness):

Move the module import to a subprocess, and call the subprocess again after recompiling, i.e. you have a Python script do_stuff.py that simply does
```
import mycext
mycext.do_stuff()
```
and you call this script using
```
subprocess.call([sys.executable, "do_stuff.py"])
```
Turn the compile-time constants in your header into variables that can be changed from Python, eliminating the need to reload the module.
Manually dlclose() the library after deleting all references to the module (a bit fragile since you don't hold all the references yourself).

Roll your own import mechanism.

Here is an example how this can be done. I wrote a minimal Python C extension mini.so, only exporting an integer called version.

>>>import ctypes>>>libdl = ctypes.CDLL("libdl.so")>>>libdl.dlclose.argtypes = [ctypes.c_void_p]>>>so = ctypes.PyDLL("./mini.so")>>>so.PyInit_mini.argtypes = []>>>so.PyInit_mini.restype = ctypes.py_object >>>mini = so.PyInit_mini()>>>mini.version
1
>>>del mini>>>libdl.dlclose(so._handle)
0
>>>del so

At this point, I incremented the version number in mini.c and recompiled.

>>>so = ctypes.PyDLL("./mini.so")>>>so.PyInit_mini.argtypes = []>>>so.PyInit_mini.restype = ctypes.py_object >>>mini = so.PyInit_mini()>>>mini.version
2

You can see that the new version of the module is used.

For reference and experimenting, here's mini.c:

#include<Python.h>staticstructPyModuleDef minimodule = {
   PyModuleDef_HEAD_INIT, "mini", NULL, -1, NULL
};

PyMODINIT_FUNC
PyInit_mini(){
    PyObject *m = PyModule_Create(&minimodule);
    PyModule_AddObject(m, "version", PyLong_FromLong(1));
    return m;
}

Solution 2:

there is another way, set a new module name, import it, and change reference to it.

Solution 3:

Update: I have now created a Python library around this approach:

Rather than using the subprocess module in Python, you can use multiprocessing. This allows the child process to inherit all of the memory from the parent (on UNIX-systems).

For this reason, you also need to be careful not to import the C extension module into the parent.

If you return a value that depends on the C extension, it might also force the C extension to become imported in the parent as it receives the return-value of the function.

import multiprocessing as mp
import sys


defsubprocess_call(fn, *args, **kwargs):
    """Executes a function in a forked subprocess"""
    
    ctx = mp.get_context('fork')
    q = ctx.Queue(1)
    is_error = ctx.Value('b', False)
    
    deftarget():
        try:
            q.put(fn(*args, **kwargs))
        except BaseException as e:
            is_error.value = True
            q.put(e)
    
    ctx.Process(target=target).start()
    result = q.get()    
    if is_error.value:
        raise result
    
    return result


defmy_c_extension_add(x, y):
    assert'my_c_extension'notin sys.modules.keys()
    # ^ Sanity check, to make sure you didn't import it in the parent processimport my_c_extension
    return my_c_extension.add(x, y)


print(subprocess_call(my_c_extension_add, 3, 4))

If you want to extract this into a decorator - for a more natural feel, you can do:

classsubprocess:
    """Decorate a function to hint that it should be run in a forked subprocess"""def__init__(self, fn):
        self.fn = fn
    def__call__(self, *args, **kwargs):
        return subprocess_call(self.fn, *args, **kwargs)


@subprocessdefmy_c_extension_add(x, y):
    assert'my_c_extension'notin sys.modules.keys()
    # ^ Sanity check, to make sure you didn't import it in the parent processimport my_c_extension
    return my_c_extension.add(x, y)


print(my_c_extension_add(3, 4))

This can be useful if you are working in a Jupyter notebook, and you want to rerun some function without rerunning all your existing cells.

Notes

This answer might only be relevant on Linux/macOS where you have a fork() system call:

Python Tutorial for Beginners