Memory issues using parfeval

Hi everyone!

Since years I am using the nice function export_fig to save figures, which I create with my scripts and functions. To speed this up I tried to do this in background using 'parfeval'. Unfortunately, for some reason workers pile up memory consumption which can be monitored e.g. with the ProcessExplorer.

function export_figP(fname, varargin)
  % same input as export_fig, but first input must be the filename, last
  % input must be the figure handle (could be improved)
  tParFeval = tic;
  tt = []; fig_1 = [];
  set(varargin{end},'Visible','off')
  myEvalStr = 'tt = parfeval(@export_fig, 0, fname';
  for n=1:nargin-2
    myEvalStr = sprintf('%s, ''%s''',myEvalStr, varargin{n});
  end
  myEvalStr = sprintf('%s, fig_1);',myEvalStr);
  n = nargin-1;
  eval(sprintf('fig_1 = varargin{%d};',n))
  eval(myEvalStr)
  afterEach(tt, @(~)evalAfterAll(tt,fig_1,fname,tParFeval), 0); 
  function evalAfterAll(tt,hh,saveStr,tParFeval) 
    if ~isempty(tt.Error)
      fprintf(2,'!!! Error saving Figure %s !!! \n',saveStr)
    else
      fprintf(2,'Figure ''%s'' saved within %.3fs  (total time %.3fs)\n',saveStr,(datenum(tt.FinishDateTime)-datenum(tt.CreateDateTime))*24*60*60, toc(tParFeval))
      close(hh)
      tt = [];
      clear all
    end
  end
end

The function can be tested with this lines (requires export_fig)

%% Test export_figP
if ~isfolder(fullfile(pwd, 'figs')); mkdir('figs');end
tic
for n = 1:1000
  % create simple figure
  fig1 = figure('Color','w','Visible','on');
  imagesc(rand(1000, 1000))
  colorbar
  % save figure in Background
  saveStr = sprintf('%04d_TestFig.png',n);
  export_figP(sprintf('figs/%s',saveStr),'-q100','-nocrop','-r300',fig1);
  fprintf('saving of %s started (%.3f)\n', saveStr, toc)
end

What do I do wrong?

How to clear the workers memory after the function call (except shuting down parallel pool)?

Probably my function 'export_figP' does something substantially wrong.

[Andre Zeug]

Is there an option to clear workers memory when no further job is processed?

p = gcp('nocreate')
p = 
 ProcessPool with properties: 

            Connected: true
           NumWorkers: 16
              Cluster: local
        AttachedFiles: {}
    AutoAddClientPath: true
          IdleTimeout: Inf (no automatic shut down)
          SpmdEnabled: true

q = p.FevalQueue
q = 
 FevalQueue with properties: 

        Number Queued: 0
       Number Running: 0          

But memory consumtion is still big after several calls of parfeval:

Answer 1

[Swastik Sarkar]

Hi @Andre Zeug,

The export_figP and evalAfterAll functions can be enhanced for readability and effective management of figures throughout their lifecycle. Below is the improved implementation:

function export_figP(fname, varargin)
    figHandle = varargin{end};
    set(figHandle, 'Visible', 'off');
    tt = parfeval(@export_fig, 0, fname, varargin{1:end-1}, figHandle);
    afterEach(tt, @(~) evalAfterAll(tt, figHandle, fname), 0);
end
function evalAfterAll(tt, figHandle, saveStr)
    if ~isempty(tt.Error)
        fprintf(2, '!!! Error saving Figure %s !!! \n', saveStr);
    else
        fprintf('Figure ''%s'' saved successfully.\n', saveStr);
    end
    
    % Close the figure and clear the task
    close(figHandle);
    delete(tt);
end

Clearing the output data of the parallel.FevalFuture object can further enhance performance. More information can be found in the following resource:

https://www.mathworks.com/matlabcentral/answers/424473-parfeval-memory-consumption-piling-up-clear-output-data#answer_342050

Hope this helps.