Batch jobs of similar data size and function files are taking different times for job completion, though all are started at same time.

1 view (last 30 days)
I have a large (n x n) data, which is an input for several dependent function files.
Each element should undergo the same process, and the communication between the elements is not necessary.
So I partitioned my data into 'z' number of zones corresponding to number of cores in local machine.
Three columns and two rows.
r_zones = 2; % row zones
c_zones = 3; % column zones
rz = round(linspace(1,sx,r_zones+1)); %rz = row zones
cz = round(linspace(1,sy,c_zones+1)); %cz = column zones
Six jobs are created for six zones with batch command.
pc = parcluster();
for i = 1:r_zones
for j = 1:c_zones
Xa((rz(i):rz(i+1)-1),((cz(j):cz(j+1)-1))) = 1;
job(jm(i,j)) = batch(pc,@com_sigma,3,{inputs});
end
end
for i = 1:r_zones
for j = 1:c_zones
wait(job(jm(i,j)));
end
end
All the jobs have same size of data ( a difference of one row or column may present) and started at the same time.
But, the finishing time of each job is different and the initial job is taking longer time. And, the CPU utilisation is not more than 70%.

Accepted Answer

Raymond Norris
Raymond Norris on 20 Aug 2021
Not sure why the 1st task isn't finishing, but here's another approach. Keep what you have, but rather than using batch, use parfeval. Change
pc = parcluster();
for i = 1:r_zones
for j = 1:c_zones
Xa((rz(i):rz(i+1)-1),((cz(j):cz(j+1)-1))) = 1;
job(jm(i,j)) = batch(pc,@com_sigma,3,{inputs});
end
end
to
pc = parcluster();
pool = gcp('nocreate');
if isempty(pool)
pool = pc.parpool(maxNumCompThreads);
end
for i = 1:r_zones
for j = 1:c_zones
Xa((rz(i):rz(i+1)-1),((cz(j):cz(j+1)-1))) = 1;
job(jm(i,j)) = parfeval(pool,@svd,3,inputs);
end
end
You're start one job (a parallel pool) and then sending individual tasks to it (with parfeval) rather than creating r_zones * c_zones jobs (with batch).

More Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!