No Module Named Numpy_pickle When Executing Script Under A Different User

I have a python script that uses sklearn joblib to load a persistent model and perform prediction. The script runs fine when I run it under my username and when some other user tri

Solution 1:

As suggested by Croad Langshan make sure you don't have a joblib version conflict/mismatch - I had exactly the same problem. The binary file was created with sklearn.externals.joblib however I was using a stand-alone joblib that I installed from the offical debian repository, this in combination with stock debian sklearn resulted in an un-unpickable binary store.

So check if you have python-joblib installed as a stand-alone package, if you do - remove it, remove sklearn and re-install sklearn from source

$ sudo apt-getremove python-joblib
$ sudo apt-getremove python-sklearn

install sklearn from source

$ git clone https://github.com/scikit-learn/scikit-learn.git$ sudo python setup.py install

*Note - a situation where the conflict is reversed is possible (original binary created with stand-alone joblib)

*A more granular solution to resolving module version conflicts/mismatch is use virtualenv, but in my case I had no incentive to keep the stand-alone joblib

Solution 2:

The load function is using the Python standard library module pickle "under the hood". That module provides a way to dump arbitrary python objects to a file. "Unpickling" that file again to load the python objects from the file back into memory requires the Python files that define the modules in which the objects' classes are defined (the same goes for functions). The directories containing those modules need to be on sys.path (say by means of being listed in environment variable PYTHONPATH).

Perhaps the pickle in question has a reference to code in module numpy_pickle (as opposed to joblib.numpy_pickle), and perhaps that is not on sys.path (even if joblib itself is). Try (before the import) running import cgitb; cgitb.enable() to see the value of module in the last stack frame.

Solution 3:

I had this same problem. I pickled a model with one user and couldn't unpickle it with a second user. The above answers didn't really help me. I believe it has something to do with the local variables saved in the pickled file and the path of the second user.

The module was trying to be loaded as:

__import__('joblib.numpy_pickle')

which results in

ImportError: No module named joblib.numpy_pickle

but if you run

__import__('sklearn.externals.joblib.numpy_pickle')

it can find it and returns

<module'sklearn'from'/python2.6/site-packages/sklearn/__init__.pyc'>

So I'm assuming the second user is trying to load the file and there are some settings in the file telling it to look in joblib.numpy_pickle and while ignoring the previously imported sklearn.externals. I didn't know how to fix this, so instead i just trained the model again with the second user and saved it. Now the second user can read the file it created.