This toolbox consists of several functions for spreading work over multiple MATLAB instances.
If you have a for loop which can be written something like:
for a = 1:size(input, 2)
output(:,a) = func(input(:,a), global_data);
then the functions in this toolbox can be used to split the for loop iterations across multiple MATLAB instances, including across multiple computing nodes/computers. This is a replacement for parfor in this use case, if you don't have the Parallel Computing Toolbox.
For the toolbox to work, its root directory needs to be on your MATLAB path at startup.
See the README at:
for more information.
These functions are in beta testing. If you have any bug reports or fixes, please raise a github issue.
batch_job_distrib works well on single PC. I'm trying to make it work in conjunction with another PC via batch_job_worker('shared_directory'). The message I'm getting is:
Could not copy batch script to host XXX
Error using start_workers (line 11)
'cat' is not recognized as an internal or external command, operable program or batch file.
Could not delete batch script on host XXX
Error using batch_job_distrib>cleanup_all (line 232)
'ssh' is not recognized as an internal or external command, operable program or batch file.
very nice work.
I had faced couple of issues in batch_job but I dont know if they were resolved.
1. matlab crashed when trying to map the memfile (at least that was what matlab was telling me), what was really going on was that with the '-automation' option matlab was trying to start an interactive session. I've resolved this with the '-nodisplay -nosplash -nodesktop', also suggested by Andreas. (around line 145 in batch_job)
2. I've workaround the variable input since in my case I can calculate maximum possible number of output and replaced the complement with 'NaNs' in the custom function.
If you can't calculate the max number of outcomes, than you can estimate with reasonable reserve, than try catch if overflows..
Hope this help a little
Hello Oliver, I need a somewhat modified version of this script. The first instance of matlab runs - performs computations and then quits. THEN (after the first instance has shutdown) the second instance should start (automatically) - perform computations (with perhaps different set of variables) and quit. Once the 2nd instance has shutdown, THEN the 3rd instance starts and so on. How (and where) to change the batch_job file? Thanks.
First, thank you a lot for this contribution.
However, I'm facing variable size output data. Like Matthew I'm hoping on cell array support.
(Similar to CELLFUN(__, 'UniformOutput', false) )
(This looks like a great piece of work! Thanks!)
It appears that this does not support non-numeric input and output, despite the markdown file stating that cell arrays can be used. I am ideally hoping to pass structs in and out (though cell arrays would also be sufficient) -- is this possible?
I would add to the command that invokes Matlab the option "-nosplash", and maybe also "-nodisplay".
Please add an example in the comment section.
Update help text
Move to github
Add the logo
Add example. Improve code.
Inspired by: Multicore - Parallel processing on multiple cores