MATLAB Crashes when Using Conda Environment Other than Base

I am having trouble using a custom conda environment in MATLAB. Here's the code I use to bootstrap a conda environment with a few packages I need from from the web:

% bootstrap a miniconda installation & python 35 environment

% choose a miniconda version, 
% !!! avoid version 4.5.12 due to openSSL DLL hell !!! 
condaUrl = 'https://repo.anaconda.com/miniconda/';
s = webread(condaUrl, weboptions('ContentType','text'));
v = unique(strcat('Miniconda3',regexp(s,'(?<=>Miniconda3)[^<>]*x86_64\.exe','match')));
vn = regexp(v,'(?<=3-)(\d*\.\d*\.\d*)','match');
verMask = cellfun(@isempty, vn);
v(verMask) = [];
vn(verMask) = [];
whichVer = listdlg('ListString',v,...
    'PromptString','Select Miniconda Version',...
    'SelectionMode','single');
if isempty(whichVer)
    % use the latest and greatest
    urlString = 'https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe';
else
    % use selected ver
    urlString = strcat(condaUrl, v{whichVer});
end

fprintf('Downloading Miniconda3 Installer\n');
targetDir = websave('miniconda3.exe', urlString);
fprintf('Miniconda3 Download Complete\n');
fprintf('Installing Miniconda3\n');
disp('start /wait "" miniconda3.exe /InstallationType=JustMe /RegisterPython=0 /S /D=%UserProfile%\Miniconda')
!start /wait "" miniconda3.exe /InstallationType=JustMe /RegisterPython=0 /S /D=%UserProfile%\Miniconda
fprintf('Miniconda3 Installation Complete\n');

% build a custom conda environment for MATLAB
pyver = 3.5;
myenv = sprintf('myenv%0.0f', pyver*10);
condaCmdPath = fullfile(getenv('USERPROFILE'),'Miniconda\Scripts\activate.bat');
minicondaRoot = fullfile(getenv('USERPROFILE'),'Miniconda');
myenvRoot = fullfile(minicondaRoot, 'envs', myenv);
matlabPythonRoot = fullfile(matlabroot, 'extern', 'engines', 'python');

% write a bat file to run in condaprompt
fptr = fopen('condaSetup.bat','w');
fprintf(fptr, 'CALL %c%s%c %c%s%c\r\n', 34, condaCmdPath, 34, 34, minicondaRoot, 34);
% fprintf(fptr, '%ccomspec%c /k %c%c%s%c %c%s%c%c\r\n', 37, 37, 34, 34, condaCmdPath, 34, 34, minicondaRoot, 34, 34);
fprintf(fptr, 'CALL conda config --add channels conda-forge\r\n');
fprintf(fptr, 'CALL conda config --set proxy_servers.http http://np1prxy801:80\r\n');
fprintf(fptr, 'CALL conda config --set proxy_servers.https https://np1prxy801:80\r\n');
fprintf(fptr, 'CALL conda create -v -y -n %s python=%0.1f numpy scipy scikit-learn sklearn-contrib-py-earth\r\n', myenv, pyver);
fprintf(fptr, 'CALL conda activate %s\r\n', myenv);
fprintf(fptr, 'cd %s\r\n', matlabPythonRoot);
fprintf(fptr, 'python setup.py build -b %s install\r\n', myenvRoot);
fclose(fptr);

% run "makefile"
dos('condaSetup.bat','-echo');

% delete the bat file
delete 'condaSetup.bat'

% add this python environment to startup.m by appending these two lines:
% !%comspec% /c ""C:\Users\HB69954\Miniconda\Scripts\activate.bat" "C:\Users\HB69954\Miniconda\envs\myenv35""
% pyversion c:\users\hb69954\miniconda\envs\myenv35\python.exe
w = which('startup.m');
fptr = fopen(w,'r');
s = fread(fptr, inf, 'uint8=>char')';
fclose(fptr);

activatePyString = '!%comspec% /c ""C:\Users\HB69954\Miniconda\Scripts\activate.bat" "C:\Users\HB69954\Miniconda\envs\myenv35""';
swapPyVersionString = 'pyversion c:\users\hb69954\miniconda\envs\myenv35\python.exe';

fptr = fopen(w,'w');
fprintf(fptr,'%s\r\n%s\r\n%s\r\n', s, activatePyString, swapPyVersionString);
fclose(fptr);

This script downloads Miniconda3, installs it, and creates a conda virtual environment called myenv35.

By default, MATLAB chooses my base environment as the python version:

>> pyversion
       version: '3.7'
    executable: 'C:\Users\HB69954\Miniconda\python.exe'
       library: 'C:\Users\HB69954\Miniconda\python37.dll'
          home: 'C:\Users\HB69954\Miniconda'
      isloaded: 1

I added these two lines to startup.m to load this python environment instead of the base environment:

!%comspec% /c ""C:\Users\HB69954\Miniconda\Scripts\activate.bat" "C:\Users\HB69954\Miniconda\envs\myenv35""
pyversion c:\users\hb69954\miniconda\envs\myenv35\python.exe

This apparently loads in MATLAB,

>> pyversion
       version: '3.5'
    executable: 'c:\users\hb69954\miniconda\envs\myenv35\python.exe'
       library: 'c:\users\hb69954\miniconda\envs\myenv35\python35.dll'
          home: 'c:\users\hb69954\miniconda\envs\myenv35'
      isloaded: 0
      

I can use python standard library functions in this environment, i.e.

>> myList = py.list([1,2,3,0,8,9,7])
myList = 
  Python list with no properties.

    [1.0, 2.0, 3.0, 0.0, 8.0, 9.0, 7.0]
>> myList.sort()
>> myList
myList = 
  Python list with no properties.

    [0.0, 1.0, 2.0, 3.0, 7.0, 8.0, 9.0]

But MATLAB crashes whenever I try to use a library that installed in the virtual environment (i.e. numpy), MATLAB closes without issuing an error or warning.

x = py.numpy.array([1,2,3])

taskmgr doesn't show matlab.exe running in the background, and no crash dump files are created in %USERPROFILE%\AppData\Local\Temp, leading me to believe its not a real crash as much as some file/function is issuing an exit/close command to matlab.exe.

1. Are python/conda environments supported in MATLAB? I can't think of a reason why they wouldn't be, but I haven't seen any official support documentation for them either.
2. Could this be related to me installing MATLAB Engine for Python in the same virtual environment?
3. Is there any other reason why swapping the python environment could be causing MATLAB to crash while running environment packages?

Answer 1

[Julian Hapke]

Try this:

load your environment in a seperate promt, have a look at the path variable (assuming windows here)

echo %PATH%

unload the environment (back to base), look at the %PATH% again and find out the differences. There should be some paths from the custom evironment added to the path variable.

Add those path inside matlab, for example with:

z

pyExec = 'C:\software\miniconda3\envs\mlpy\python.exe';
pyRoot = fileparts(pyExec);
p = getenv('PATH');
p = strsplit(p, ';');
addToPath = {
    pyRoot
    fullfile(pyRoot, 'Library', 'mingw-w64', 'bin')
    fullfile(pyRoot, 'Library', 'usr', 'bin')
    fullfile(pyRoot, 'Library', 'bin')
    fullfile(pyRoot, 'Scripts')
    fullfile(pyRoot, 'bin')
    };
p = [addToPath(:); p(:)];
p = unique(p, 'stable');
p = strjoin(p, ';');
setenv('PATH', p);

after that MATLAB should be able to use packages from that conda environment.

[Peter Cook]

Apologies for the late response Julian,

I took your suggestion and see that python appends the following directories to path when I activate my conda environment and removes them when I deactivate :

'C:\Users\HB69954\Miniconda\envs\myenv35'
'C:\Users\HB69954\Miniconda\envs\myenv35\Library\bin'
'C:\Users\HB69954\Miniconda\envs\myenv35\Library\mingw-w64\bin'
'C:\Users\HB69954\Miniconda\envs\myenv35\Library\usr\bin'
'C:\Users\HB69954\Miniconda\envs\myenv35\Scripts'
'C:\Users\HB69954\Miniconda\envs\myenv35\bin'

and when I activate my python base environment, I get these:

'C:\Users\HB69954\Miniconda\Library\mingw-w64\bin'
'C:\Users\HB69954\Miniconda\Library\usr\bin'
'C:\Users\HB69954\Miniconda\bin'
'C:\Program Files\dotnet\'

I do not wish to append the environment specific python variables to PATH with setenv (or py.sys.path() ) permanently, as I use other python environments outside of MATLAB and don't want path shadowing.

But looking into that brought something interesting to light. I noticed multiple entries in path to pywin32_system32. I then looked into a couple python 3 versions of pywin32.pth and found an interesting difference.

Python 3.6:

# .pth file for the PyWin32 extensions
win32
win32\lib
Pythonwin
# Entries needed for a "portable" installations, where the post_install script
# isn't run, which would normally copy the pywin32 core DLL files to either
# the top of the python directory.
# We just stick the source of these DLLs directly on the PATH.
import os;os.environ["PATH"]+=(';'+os.path.join(sitedir,"pywin32_system32"))

Python 3.7:

# .pth file for the PyWin32 extensions
win32
win32\lib
Pythonwin
# This breaks any software involving a feedback loop involving PATH and Python
# One example of such software is the Anaconda Distribition's conda package
# manager, so we do not do it. We fixed this in a slightly better way anyway.
# Entries needed for a "portable" installations, where the post_install script
# isn't run, which would normally copy the pywin32 core DLL files to either
# the top of the python directory.
# We just stick the source of these DLLs directly on the PATH.
# import os;os.environ["PATH"]+=(';'+os.path.join(sitedir,"pywin32_system32"))

Two telling things from the comments in this second find file: 1. We find the cause of the crash and 2. Python 3.7 does not append this (command is muted) to the path whereas python < 3.7 does (possibly also explaining the addition of C:\Program Files\dotnet, a directly full of MSVC DLLs, to path). Since my base environment is python 3.7 this explains the sometimes working, sometimes breaking behavior seen in MATLAB when swapping to the 3.5 environment.

[Julian Hapke]

Adding to %PATH% with setenv should only modify the PATH in the current MATLAB Session. Outside of MATLAB everything should remain default, so I don't see a problem there.

[Fahad Raza]

@Julian Thank you. It worked absolutely fine and you saved my day.

[Seth Wagenman]

The only way to make sure this works is to run a Python script importing a library/module not in the base conda environment. But it definitely works!

[JB676]

Wow, what a great pickup. I had conda-installed numpy, and it was crashing. Changing that to a pip-install, allows it to run. Never would have guessed to check that.

Answer 2

[Seth Wagenman]

For a conda environment named "useFromMATLAB", the following code works in Windows 10/Anaconda 3. Note that if you are debugging Python at the same time, and making changes to "some_awesome_python_module," you have to reload it every time (code below starting with clear classes is how to do that).

py_root_useFromMATLAB = fileparts(C:\anaconda_3\envs\useFromMATLAB\_conda.exe);
ENV = getenv('PATH');
ENV = strsplit(ENV, ';');
items_to_add_to_path = {
    fullfile(py_root_useFromMATLAB, 'Library', 'mingw-w64', 'bin')
    fullfile(py_root_useFromMATLAB, 'Library', 'usr', 'bin')
    fullfile(py_root_useFromMATLAB, 'Library', 'bin')
    fullfile(py_root_useFromMATLAB, 'Scripts')
    };
ENV = [items_to_add_to_path(:); ENV(:)];
ENV = unique(ENV, 'stable');
ENV = strjoin(ENV, ';');
setenv('PATH', ENV);

clear classes
module_to_load = 'some_awesome_python_module';
python_module_to_use = py.importlib.import_module(module_to_load);
py.importlib.reload(python_module_to_use);
% Now you can use it like output = py.some_awesome_python_module.some_awesome_python_function(input)

[Seth Wagenman]

https://www.mathworks.com/help/matlab/matlab_external/call-modified-python-module.html

Answer 3

[Tucker Downs]

The obvious feature / solution would be for Mathworks to just respect the user environment when people want to call a python script from matlab. Getting a simple script to run with my anaconda environment should not require such work arounds when I can open VSCode or terminal and run the exact same script in less than 10 secconds.

Extremely dissapointed in this right now, as much as I love Matlab it's [stuff] like this that makes me want to leave the platform and go full time python. Even with the pain of losing MW support and documentation. The core features are just not keeping up with other development stratagies.

[Walter Roberson]

"The obvious feature / solution would be for Mathworks to just respect the user environment when people want to call a python script from matlab."

That would probably not work in newer MacOS. The security model for newer MacOS is that applications must provide their own copies of third-party libraries that they depend on, rather than using a user-installed environment. The claim is that this increases security by firewalling different apps from each other -- that if (for example0 MATLAB provides its own Python, then a malicious MATLAB program would not be able to change the Python state for a different App that used Python .

Apple is pretty serious about this change in their security model. For example it was made clear to NVidia that even if NVidia provided their own graphics drivers, that every program would need to include its own copy of the driver, that it would not be permitted for users to install an NVidia graphics driver for shared use system-wide.

[dleal]

Now that you talk about MacOS on an M1.. I've been trying for a few days to run an anaconda environment on MATLAB, but it crashes similarly as described above. Python loads correctly, but if i load another library like pandas, it crashes....I have tried adding all folders and subfolders of the environment root as follows:

[status, result]=system('source ~/opt/anaconda3/bin/activate matpy2');
conda_env_dir = '/Users/diegoleal/opt/anaconda3/envs/matpy2';
dirs_ = splitlines(result);
p = getenv('PATH');
p = strsplit(p, ':');
p = [dirs_(:); p(:)];
p = unique(p, 'stable');
p = strjoin(p, ':');
setenv('PATH', p);

it does work with base, though, that it:

pyenv('Version', '/Users/diegoleal/opt/anaconda3/bin/python3')
py.importlib.import_module('numpy')

causes no problems

What else should I try?

[Walter Roberson]

Sorry, I have no experience with Python or with M1 (well, other than I recently did a backup of one)

[Tucker Downs]

Hi @Walter Roberson thanks for your insightful comment.

It will be interesting to see how that affects vscode. I'm going to guess that I'll see the same functionality in code as I always have, in which case there won't really an excuse for a code development product like MATLAB desktop.

[Majid Khalili]

@Tucker Downs did you manage to make it work for activating an envirioment from anaconda in Matlab?

[raym]

% I tried this, it does not work on some modules like anndata.

% Define the root of the Conda environment
condaEnvRoot = 'D:\GreenSoft\Anaconda3\envs\annadata';
% Get the current PATH environment variable
currentPath = strsplit(getenv('PATH'),';')';disp(getenv('PATH'));
% Define the paths to add from the Conda environment
pathsToAdd = {
    condaEnvRoot;
    fullfile(condaEnvRoot, 'Library', 'mingw-w64','bin');
    fullfile(condaEnvRoot, 'Library', 'usr','bin');
    fullfile(condaEnvRoot, 'Library', 'bin');
    fullfile(condaEnvRoot, 'Scripts');
    fullfile(condaEnvRoot, 'bin');
    'D:\GreenSoft\Anaconda3\condabin';
    
    fullfile(condaEnvRoot, 'Library', 'site-packages','scipy.libs');
    fullfile(condaEnvRoot, 'Library', 'site-packages','pandas','_libs','window');
    fullfile(condaEnvRoot, 'Library', 'site-packages','numpy');
    fullfile(condaEnvRoot, 'Library', 'site-packages','llvmlite','binding');
    fullfile(condaEnvRoot, 'Library', 'site-packages','h5py');
    'D:\GreenSoft\Anaconda3\Scripts';
    'D:\GreenSoft\Anaconda3\DLLs';
    'D:\GreenSoft\Anaconda3\Lib\site-packages';
    'D:\GreenSoft\Anaconda3';
    };
]newPath = [pathsToAdd; currentPath;];
newPath = unique(newPath, 'stable');
setenv('PATH',strjoin(newPath,';'));
disp(getenv('PATH'));
pyExec = fullfile(condaEnvRoot, 'python.exe');
pyversion(pyExec);
try
    py.importlib.import_module('numpy');
    disp('Numpy loaded successfully from the Conda environment.');
catch e
    disp('Error loading Numpy:');
    disp(e.message);
end
pyversion
       version: '3.8'
    executable: 'D:\GreenSoft\Anaconda3\envs\annadata\python.exe'
       library: 'D:\GreenSoft\Anaconda3\envs\annadata\python38.dll'
          home: 'D:\GreenSoft\Anaconda3\envs\annadata'
      isloaded: 1
pandas = py.importlib.import_module('pandas');%ok
numpy = py.importlib.import_module('numpy'); %ok
anndata = py.importlib.import_module('anndata'); %Python Error: ImportError: DLL load failed while importing _errors: cannot find specified programme。

[raym]

The culprit is in this file:

anndata = py.importlib.import_module('h5py');

% D:\GreenSoft\Anaconda3\envs\annadata\Lib\site-packages\h5py\__init__.py

The culprit is the syntax "from . import ...".

I think the dot should be changed to something else to make it work .I also cannot find _error.py anywhere inside h5py folder. But import h5py in conda cmd window works.

# --- Library setup -----------------------------------------------------------

# When importing from the root of the unpacked tarball or git checkout,
# Python sees the "h5py" source directory and tries to load it, which fails.
# We tried working around this by using "package_dir" but that breaks Cython.
try:
    from . import _errors
    except ImportError:
    import os.path as _op
    if _op.exists(_op.join(_op.dirname(__file__), '..', 'setup.py')):
        raise ImportError("You cannot import h5py from inside the install directory.\nChange to another directory first.")
    else:
        raise
        
        from . import version