File Exchange

image thumbnail

Batch Job

version 1.5.0.0 (26 KB) by Oliver Woodford
Toolbox to spread batch work over multiple MATLAB instances, even on other PCs

8 Downloads

Updated 18 May 2018

GitHub view license on GitHub

This toolbox consists of several functions for spreading work over multiple MATLAB instances.
If you have a for loop which can be written something like:
for a = 1:size(input, 2)
output(:,a) = func(input(:,a), global_data);
end

then the functions in this toolbox can be used to split the for loop iterations across multiple MATLAB instances, including across multiple computing nodes/computers. This is a replacement for parfor in this use case, if you don't have the Parallel Computing Toolbox.

For the toolbox to work, its root directory needs to be on your MATLAB path at startup.

See the README at:
https://github.com/ojwoodford/batch_job/blob/master/README.md
for more information.

These functions are in beta testing. If you have any bug reports or fixes, please raise a github issue.

Comments and Ratings (8)

grega

batch_job_distrib works well on single PC. I'm trying to make it work in conjunction with another PC via batch_job_worker('shared_directory'). The message I'm getting is:

Could not copy batch script to host XXX
Error using start_workers (line 11)
'cat' is not recognized as an internal or external command, operable program or batch file.

AND

Could not delete batch script on host XXX
Error using batch_job_distrib>cleanup_all (line 232)
'ssh' is not recognized as an internal or external command, operable program or batch file.

Please suggestions?

massa64

Hi,
very nice work.
I had faced couple of issues in batch_job but I dont know if they were resolved.
1. matlab crashed when trying to map the memfile (at least that was what matlab was telling me), what was really going on was that with the '-automation' option matlab was trying to start an interactive session. I've resolved this with the '-nodisplay -nosplash -nodesktop', also suggested by Andreas. (around line 145 in batch_job)
2. I've workaround the variable input since in my case I can calculate maximum possible number of output and replaced the complement with 'NaNs' in the custom function.
If you can't calculate the max number of outcomes, than you can estimate with reasonable reserve, than try catch if overflows..
Hope this help a little
Regards

b

Hello Oliver, I need a somewhat modified version of this script. The first instance of matlab runs - performs computations and then quits. THEN (after the first instance has shutdown) the second instance should start (automatically) - perform computations (with perhaps different set of variables) and quit. Once the 2nd instance has shutdown, THEN the 3rd instance starts and so on. How (and where) to change the batch_job file? Thanks.

David T_

First, thank you a lot for this contribution.

However, I'm facing variable size output data. Like Matthew I'm hoping on cell array support.
(Similar to CELLFUN(__, 'UniformOutput', false) )

Matthew

(This looks like a great piece of work! Thanks!)

It appears that this does not support non-numeric input and output, despite the markdown file stating that cell arrays can be used. I am ideally hoping to pass structs in and out (though cell arrays would also be sufficient) -- is this possible?

Very nice.
I would add to the command that invokes Matlab the option "-nosplash", and maybe also "-nodisplay".

Andreas

Please add an example in the comment section.

Updates

1.5.0.0

Update help text

1.5.0.0

Update description

1.4.0.0

Move to github

1.3.0.0

Bug fix

1.2.0.0

Add the logo

1.1.0.0

Add example. Improve code.

MATLAB Release Compatibility
Created with R2013a
Compatible with any release
Platform Compatibility
Windows macOS Linux

Discover Live Editor

Create scripts with code, output, and formatted text in a single executable document.


Learn About Live Editor