Importing custom python module fails

I am using Matlab2014b and try to import python packages. I am on a linux computer and executing the command py.numpy.* works and py.importlib.import_module('numpy') gives me the correct output. However, numpy lives in my /usr/lib/pymodules/python2.7/ directory, which according to the output of py.sys.path is in the search directory, so the import should work.

However, I have another package that I would like to include, which is in /usr/lib/python2.7/dist-packages/. Again, according to py.sys.path, this directory is included in the search path, however, importing the package fails ("Import argument 'py.fabio' cannot be found or cannot be imported."). I hope someone has an advice how to solve this problem.

PS: pyversion returns my current installation (2.7) and shows isloaded: 1

PPS: Adding the directory to the search path, as outlined in http://www.mathworks.com/help/matlab/matlab_external/call-python-from-matlab.html has failed.

PPPS: When I try to get feedback from py.importlib.import_module, Matlab crashes.

[Robert Snoeberger]

To clarify, what do you mean by "Matlab crashes" and what was the full command given? I'm using Python 3.4 and it works for me:

>> py.importlib.import_module('fabio')

ans = 

    Python module with properties:

                    version: [1x5 py.str]
                 HiPiCimage: [1x1 py.module]
                COMPRESSORS: [1x1 py.dict]
                 openheader: [1x1 py.function]
               pilatusimage: [1x1 py.module]
                   edfimage: [1x1 py.module]
...

Note: I truncated the display of ans because it is quite long.

[Fernando Bechtelar]

I was trying the following, to add the package fabio (I/O library for 2D images, see https://pypi.python.org/pypi/fabio) to the path:

modpath = '/usr/lib/python2.7/dist-packages/fabio';
% follow the approach from the matlab documentation center
P = py.sys.path;
  if count(P,modpath) == 0
  insert(P,int32(0),modpath);
end

and then either

import py.fabio % fails

or

py.importlib.import_module('fabio') % crashes

The error message is: Matlab has encountered an internal problem and needs to close. Under Details in the stack trace the directory /usr/lib/python2.7.so appears frequently, but I do not know, what that means.

[Robert Snoeberger]

/usr/bin/python2.7.so is the shared library for Python. It might be helpful to attach the stack trace.

[Fernando Bechtelar]

This is the stack trace when I try to include the fabio package. I tried to skim it a bit to make to post not too cluttered.

Stack Trace (from fault):
[  0] /lib/x86_64-linux-gnu/libc.so.6+00205157 gsignal+00000053
[  1] /lib/x86_64-linux-gnu/libc.so.6+00218080 abort+00000384
[  2] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libhdf5.so.8+00224629 H5check_version+00000325
[  3] /usr/lib/python2.7/dist-packages/h5py/h5f.so+00042095 inith5f+00006975
[  4] /usr/lib/libpython2.7.so+01579985 _PyImport_LoadDynamicModule+00000193
[  5] /usr/lib/libpython2.7.so+01109372
[  6] /usr/lib/libpython2.7.so+01110002
[  7] /usr/lib/libpython2.7.so+01110925 PyImport_ImportModuleLevel+00000605
[  8] /usr/lib/libpython2.7.so+00666527
[  9] /usr/lib/libpython2.7.so+01269118 PyObject_Call+00000078
[ 10] /usr/lib/libpython2.7.so+01271431 PyEval_CallObjectWithKeywords+00000071
[ 11] /usr/lib/libpython2.7.so+00306307 PyEval_EvalFrameEx+00004915
[ 12] /usr/lib/libpython2.7.so+00328424 PyEval_EvalCodeEx+00002120
[ 13] /usr/lib/libpython2.7.so+00328738 PyEval_EvalCode+00000050
[ 14] /usr/lib/libpython2.7.so+00685773 PyImport_ExecCodeModuleEx+00000173
...
[ 19] /usr/lib/libpython2.7.so+01110685 PyImport_ImportModuleLevel+00000365
[ 20] /usr/lib/libpython2.7.so+00666527
[ 21] /usr/lib/libpython2.7.so+01269118 PyObject_Call+00000078
[ 22] /usr/lib/libpython2.7.so+01271431 PyEval_CallObjectWithKeywords+00000071
[ 23] /usr/lib/libpython2.7.so+00306307 PyEval_EvalFrameEx+00004915
[ 24] /usr/lib/libpython2.7.so+00328424 PyEval_EvalCodeEx+00002120
[ 25] /usr/lib/libpython2.7.so+00328738 PyEval_EvalCode+00000050
[ 26] /usr/lib/libpython2.7.so+00685773 PyImport_ExecCodeModuleEx+00000173
...
[ 30] /usr/lib/libpython2.7.so+01110925 PyImport_ImportModuleLevel+00000605
[ 31] /usr/lib/libpython2.7.so+00666527
[ 32] /usr/lib/libpython2.7.so+01269118 PyObject_Call+00000078
[ 33] /usr/lib/libpython2.7.so+01271431 PyEval_CallObjectWithKeywords+00000071
[ 34] /usr/lib/libpython2.7.so+00306307 PyEval_EvalFrameEx+00004915
[ 35] /usr/lib/libpython2.7.so+00328424 PyEval_EvalCodeEx+00002120
[ 36] /usr/lib/libpython2.7.so+00328738 PyEval_EvalCode+00000050
[ 37] /usr/lib/libpython2.7.so+00685773 PyImport_ExecCodeModuleEx+00000173
...
[ 41] /usr/lib/libpython2.7.so+01110925 PyImport_ImportModuleLevel+00000605
[ 42] /usr/lib/libpython2.7.so+00666527
[ 43] /usr/lib/libpython2.7.so+01269118 PyObject_Call+00000078
[ 44] /usr/lib/libpython2.7.so+01271431 PyEval_CallObjectWithKeywords+00000071
[ 45] /usr/lib/libpython2.7.so+00306307 PyEval_EvalFrameEx+00004915
[ 46] /usr/lib/libpython2.7.so+00328424 PyEval_EvalCodeEx+00002120
[ 47] /usr/lib/libpython2.7.so+00328738 PyEval_EvalCode+00000050
[ 48] /usr/lib/libpython2.7.so+00685773 PyImport_ExecCodeModuleEx+00000173
...
[ 53] /usr/lib/libpython2.7.so+01110685 PyImport_ImportModuleLevel+00000365
[ 54] /usr/lib/libpython2.7.so+00666527
[ 55] /usr/lib/libpython2.7.so+00324642 PyEval_EvalFrameEx+00023250
[ 56] /usr/lib/libpython2.7.so+00328424 PyEval_EvalCodeEx+00002120
[ 57] /usr/lib/libpython2.7.so+00682263
[ 58] /usr/lib/libpython2.7.so+01269118 PyObject_Call+00000078
[ 59] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwpycli.so+00299020
[ 60] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwpycli.so+00289121
[ 61] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwpycli.so+00425954
[ 62] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwmcos_impl.so+01540515
...
[ 66] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwmcos_impl.so+02046678
[ 67] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwm_lxe.so+09159856
...
[ 81] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwbridge.so+00168636
[ 82] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwbridge.so+00171767 _Z8mnParserv+00000823
[ 83] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwmcr.so+00686927 _ZN11mcrInstance30mnParser_on_interpreter_threadEv+00000031
[ 84] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwmcr.so+00607299
[ 85] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwmcr.so+00608825 _ZN5boost6detail11task_objectIvNS_3_bi6bind_tIvPFvRKNS_8functionIFvvEEEENS2_5list1INS2_5valueIS6_EEEEEEE6do_runEv+00000025
[ 86] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwmcr.so+00614215 _ZN5boost6detail9task_baseIvE3runEv+00000071
[ 87] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwmcr.so+00614311
[ 88] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwmcr.so+00594682
[ 89] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwuix.so+00313014
[ 90] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwuix.so+00206946
[ 91] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwservices.so+03002591
...
[ 94] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwservices.so+03012684 _Z25svWS_ProcessPendingEventsiib+00000092
...
[ 97] /mntdirect/_sware/com/matlab_2015b/bin/glnxa64/libmwmcr.so+00516077
[ 98] /lib/x86_64-linux-gnu/libpthread.so.0+00027472
[ 99] /lib/x86_64-linux-gnu/libc.so.6+00899421 clone+00000109

Answer 1

[Robert Snoeberger]

Based on the stack trace, your issue looks like a library conflict with libhdf5.so. It seems like the libhdf5 that is in the stack trace is a library that is installed with MATLAB; I am assuming this based on the '/matlab_2015b/bin/glnxa64/' in the path to the library. I would guess that the appropriate version of libhdf5.so for Python should be located at /usr/lib/libhdf5.so. Try the following command to look for the correct libhdf5.so.

ldd /usr/lib/python2.7/dist-packages/h5py/h5f.so | grep libhdf5

My hunch about the library conflict is further confirmed by the fact that the libhdf5.so function that is executing is called H5check_version (line [2] of the stack trace). The documentation I found for that function says, "If this check fails, H5check_version causes the application to abort (by means of a standard C abort() call)..." [1].

Assuming you aren't using MATLAB's interface to HDF5, you should be able to use LD_PRELOAD to load the correct version of libhdf5.so [2,3].

References

1. https://www.hdfgroup.org/HDF5/doc1.6/RM_H5.html#Library-VersCheck
2. http://man7.org/linux/man-pages/man8/ld.so.8.html -- Search for "LD_PRELOAD". It should say, "A list of additional, user-specified, ELF shared objects to be loaded before all others..."
3. http://stackoverflow.com/questions/426230/what-is-the-ld-preload-trick

[Fernando Bechtelar]

The output of ldd was the following:

        libhdf5_serial.so.8 => /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8 (0x00007feb0422c000)
        libhdf5_serial_hl.so.8 => /usr/lib/x86_64-linux-gnu/libhdf5_serial_hl.so.8 (0x00007feb03ffb000)

I am quite sure, your answer is correct. The only problem I have now is, that if I set LD_PRELOAD (I guess, I would have to do something like $ LD_PRELOAD = /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.8) I guess this would affect all other users of this computer. I am not sure, if that is such a great idea.

The work around I will use for the moment is to use a different computer that does not use the hdf5 interface of Matlab.

[Robert Snoeberger]

Another workaround to try is to set the dlopen flag to RTLD_NOW | RTLD_DEEPBIND [1], which can be done with Python's sys.setdlopenflags [2].

Example

>> RTLD_NOW = 2;
>> RTLD_DEEPBIND = 8;
>> flag = bitor(RTLD_NOW, RTLD_DEEPBIND); % RTLD_NOW | RTLD_DEEPBIND
>> py.sys.setdlopenflags(int32(flag));
>> py.importlib.import_module('fabio');

References

1. http://linux.die.net/man/3/dlopen -- Search for "RTLD_DEEPBIND".
2. https://docs.python.org/2/library/sys.html#sys.setdlopenflags

[Fernando Bechtelar]

Wonderful, that worked. Thank you for your great support!

[Bingzhao Shan]

Hi, I'm having Similar issue on Windows10. I tried the previous method but it seems that they don't work on Windows. Is there any suggestion? Thanks!

[Michael Thieme]

I'm also having this issue on Windows 10, it would be great if someone is able to find a workaround!

[Michael Thieme]

All,

I found a solution to this problem. My problem was related to tensorflow/keras rather than fabio. This thread (https://github.com/NeurodataWithoutBorders/api-python/issues/28) led me to the solution. Basically my order of operations was:

• conda uninstall hdf5
• conda uninstall h5py
• pip uninstall hdf5
• pip uninstall h5py
• conda install hdf5=1.8.12 h5py=2.7.0 -c jonc
• pip install keras
• pip install tensorflow-gpu

Answer 2

[Rafael Rebouças]

My solution:

% Simple way to load your library
copyfile(my_library_file_path, pwd,  'f');
% It's necessary to load library
py.my_library.any_method_or_function();
delete 'my_library.py';

Now, your Python library is loaded to use in any directory.

[Ronan Fleming]

https://nl.mathworks.com/help/releases/R2021a/matlab/matlab_external/out-of-process-execution-of-python-functionality.html