Using sim Function Within parfor

Note

Using sim function within parfor loop is no longer recommended. For running parallel simulations, use parsim. Please see Run Parallel Simulations.

Overview of Calling sim from Within parfor

The parfor command allows you to run parallel (simultaneous) Simulink^® simulations of your model (design). In this context, parallel runs mean multiple model simulations at the same time on different workers. Calling sim from within a parfor loop often helps for performing multiple simulation runs of the same model for different inputs or for different parameter settings. For example, you can save simulation time performing parameter sweeps and Monte Carlo analyses by running them in parallel. Note that running parallel simulations using parfor does not currently support decomposing your model into smaller connected pieces and running the individual pieces simultaneously on multiple workers.

Normal, Accelerator, and Rapid Accelerator simulation modes are supported by sim in parfor. (See Choosing a Simulation Mode for details on selecting a simulation mode and Design Your Model for Effective Acceleration for optimizing simulation run times.) For other simulation modes, you need to address any workspace access issues and data concurrency issues to produce useful results. Specifically, the simulations need to create separately named output files and workspace variables. Otherwise, each simulation overwrites the same workspace variables and files, or can have collisions trying to write variables and files simultaneously.

For information on code regeneration and parameter handling in Rapid Accelerator mode, see Parameter Tuning in Rapid Accelerator Mode.

Also, see parfor (Parallel Computing Toolbox).

Note

If you open models inside a parfor statement, close them again using bdclose all to avoid leaving temporary files behind.

sim in parfor with Normal Mode

This code fragment shows how you can use sim and parfor in Normal mode. Save changes to your model before simulating in parfor. The saved copy of your model is distributed to parallel workers when simulating in parfor.

% 1) Load model and initialize the pool.
openExample('sldemo_suspn_3dof');
model = 'sldemo_suspn_3dof';
load_system(model);
parpool;

% 2) Set up the iterations that we want to compute.
Cf                  = evalin('base', 'Cf');
Cf_sweep            = Cf*(0.05:0.1:0.95);
iterations          = length(Cf_sweep);
simout(iterations)  = Simulink.SimulationOutput;

% 3) Need to switch all workers to a separate tempdir in case 
% any code is generated for instance for StateFlow, or any other 
% file artifacts are  created by the model.
spmd
    % Setup tempdir and cd into it
    currDir = pwd;
    addpath(currDir);
    tmpDir = tempname;
    mkdir(tmpDir);
    cd(tmpDir);
    % Load the model on the worker
    load_system(model);
end

% 4) Loop over the number of iterations and perform the
% computation for different parameter values.
parfor idx=1:iterations   
    set_param([model '/Road-Suspension Interaction'],'MaskValues',...
        {'Kf',num2str(Cf_sweep(idx)),'Kr','Cr'});
    simout(idx) = sim(model, 'SimulationMode', 'normal');
end

% 5) Switch all of the workers back to their original folder.
spmd
    cd(currDir);
    rmdir(tmpDir,'s');
    rmpath(currDir);
    close_system(model, 0);
end

close_system(model, 0);
delete(gcp('nocreate'));

sim in parfor with Normal Mode and MATLAB Parallel Server Software

This code fragment is identical to the one in sim in parfor with Normal Mode

. Modify it as follows for using sim and parfor in Normal mode:

• In item 1, modify the parpool command as follows to create an object and use it to call a cluster name.

  p = parpool('clusterProfile');
  % 'clusterProfile' is the name of the distributed cluster

• In item 1, find files on which the model depends and attach those files to the model for distribution to cluster workers on remote machines.

  files = dependencies.fileDependencyAnalysis(modelName);
  p.addAttachedFiles(files);

• If you do not have a MATLAB^® Parallel Server™ cluster, use your local cluster. For more information, see Discover Clusters and Use Cluster Profiles (Parallel Computing Toolbox).

Start your cluster before running the code.

% 1) Load model and initialize the pool.
openExample('sldemo_suspn_3dof');
model = 'sldemo_suspn_3dof';
load_system(model);
parpool;

% 2) Set up the iterations that we want to compute.
Cf                  = evalin('base', 'Cf');
Cf_sweep            = Cf*(0.05:0.1:0.95);
iterations          = length(Cf_sweep);
simout(iterations)  = Simulink.SimulationOutput;

% 3) Need to switch all workers to a separate tempdir in case 
% any code is generated for instance for StateFlow, or any other 
% file artifacts are  created by the model.
spmd
    % Setup tempdir and cd into it
    addpath(pwd);
    currDir = pwd;
    addpath(currDir);
    tmpDir = tempname;
    mkdir(tmpDir);
    cd(tmpDir);
    % Load the model on the worker
    load_system(model);
end

% 4) Loop over the number of iterations and perform the
% computation for different parameter values.
parfor idx=1:iterations   
    set_param([model '/Road-Suspension Interaction'],'MaskValues',...
        {'Kf',num2str(Cf_sweep(idx)),'Kr','Cr'});
    simout(idx) = sim(model, 'SimulationMode', 'normal');
end

% 5) Switch all of the workers back to their original folder.
spmd
    cd(currDir);
    rmdir(tmpDir,'s');
    rmpath(currDir);
    close_system(model, 0);
end

close_system(model, 0);
delete(gcp('nocreate'));

sim in parfor with Rapid Accelerator Mode

Running Rapid Accelerator simulations in parfor combines speed with automatic distribution of a prebuilt executable to the parfor workers. As a result, this mode eliminates duplication of the update diagram phase.

To run parallel simulations in Rapid Accelerator simulation mode using the sim and parfor commands:

• Configure the model to run in Rapid Accelerator simulation mode.

• Save changes to your model before simulating in parfor. The saved copy of your model is distributed to parallel workers when simulating in parfor.

• Ensure that the Rapid Accelerator target is already built and up to date.

• Disable the Rapid Accelerator target up-to-date check by setting the sim command option RapidAcceleratorUpToDateCheck to 'off'.

To satisfy the second condition, you can change parameters only between simulations that do not require a model rebuild. In other words, the structural checksum of the model must remain the same. Hence, you can change only tunable block diagram parameters and tunable run-time block parameters between simulations. For a discussion on tunable parameters that do not require a rebuild subsequent to their modifications, see Determine If the Simulation Will Rebuild.

To disable the Rapid Accelerator target up-to-date check, use the sim command, as shown in this sample.

parpool;
% Load the model and set parameters
model = 'vdp';
load_system(model);
% Build the Rapid Accelerator target
rtp = Simulink.BlockDiagram.buildRapidAcceleratorTarget(model);
% Run parallel simulations
parfor i=1:4
   simOut{i} = sim(model,'SimulationMode', 'rapid',...
               'RapidAcceleratorUpToDateCheck', 'off',...
               'SaveTime', 'on',...
               'StopTime', num2str(10*i));
   close_system(model, 0);
end

close_system(model, 0);
delete(gcp('nocreate'));

In this example, the call to the buildRapidAcceleratorTarget function generates code once. Subsequent calls to sim with the RapidAcceleratorUpToDateCheck option off guarantees that code is not regenerated. Data concurrency issues are thus resolved.

When you set RapidAcceleratorUpToDateCheck to 'off', changes that you make to block parameter values in the model (for example, by using block dialog boxes, by using the set_param function, or by changing the values of MATLAB variables) do not affect the simulation. Instead, use RapidAcceleratorParameterSets to pass new parameter values directly to the simulation.

Workspace Access Issues

Workspace Access for MATLAB worker sessions.  By default, to run sim in parfor, a parallel pool opens automatically, enabling the code to run in parallel. Alternatively, you can also first open MATLAB workers using the parpool command. The parfor command then runs the code within the parfor loop in these MATLAB worker sessions. The MATLAB workers, however, do not have access to the workspace of the MATLAB client session where the model and its associated workspace variables have been loaded. Hence, if you load a model and define its associated workspace variables outside of and before a parfor loop, then neither is the model loaded, nor are the workspace variables defined in the MATLAB worker sessions where the parfor iterations are executed. This is typically the case when you define model parameters or external inputs in the base workspace of the client session. These scenarios constitute workspace access issues.

Transparency Violation.  When you run sim in parfor with srcWorkspace set to current, Simulink uses the parfor workspace, which is a transparent workspace. Simulink then displays an error for transparency violation. For more information on transparent workspaces, see Ensure Transparency in parfor-Loops or spmd Statements (Parallel Computing Toolbox) .

Data Dictionary Access.  When a model is linked to a data dictionary (see What Is a Data Dictionary?), to write code in parfor that accesses a variable or object that you store in the dictionary, you must use the functions Simulink.data.dictionary.setupWorkerCache and Simulink.data.dictionary.cleanupWorkerCache to prevent access issues. For an example, see Sweep Variant Control Using Parallel Simulation.

Resolving Workspace Access Issues

When a Simulink model is loaded into memory in a MATLAB client session, it is only visible and accessible in that MATLAB session; it is not accessible in the memory of the MATLAB worker sessions. Similarly, the workspace variables associated with a model that are defined in a MATLAB client session (such as parameters and external inputs) are not automatically available in the worker sessions. You must therefore ensure that the model is loaded and that the workspace variables referenced in the model are defined in the MATLAB worker session by using the following two methods.

• In the parfor loop, use the sim command to load the model and to set parameters that change with each iteration. (Alternative: load the model and then use the g(s)et_param command(s) to set the parameters in the parfor loop)

• In the parfor loop, use the MATLAB evalin and assignin commands to assign data values to variables.

Alternatively, you can simplify the management of workspace variables by defining them in the model workspace. These variables will then be automatically loaded when the model is loaded into the worker sessions. There are, however, limitations to this method. For example, you cannot store signal objects that use a storage class other than Auto in a model workspace. For a detailed discussion on the model workspace, see Model Workspaces.

Specifying Parameter Values Using the sim Command.  Use the sim command in the parfor loop to set parameters that change with each iteration.

%Specifying Parameter Values Using the sim Command
    
model = 'vdp';
load_system(model)
    
%Specifying parameter values. 
paramName = 'StopTime';
paramValue = {'10', '20', '30', '40'};
    
% Run parallel simulations
parfor i=1:4
    simOut{i} = sim(model, ...
                    paramName, paramValue{i}, ...
                    'SaveTime', 'on'); %#ok
end
    
close_system(model, 0);

An equivalent method is to load the model and then use the set_param command to set the paramName in the parfor loop.

Specifying Variable Values Using the assignin Command.  You can pass the values of model or simulation variables to the MATLAB workers by using the assignin or the evalin command. The following example illustrates how to use this method to load variable values into the appropriate workspace of the MATLAB workers.

parfor i = 1:4
    assignin('base', 'extInp', paramValue{i})%#ok
    % 'extInp' is the name of the variable in the base 
    % workspace which contains the External Input data
    simOut{i} = sim(model, 'ExternalInput', 'extInp'); %#ok
end

Sweep Variant Control Using Parallel Simulation.  To use parallel simulation to sweep a variant control (a Simulink.Parameter object whose value influences the variant condition of a Simulink.Variant object) that you store in a data dictionary, use this code as a template. Change the names and values of the model, data dictionary, and variant control to match your application.

To sweep block parameter values or the values of workspace variables that you use to set block parameters, use Simulink.SimulationInput objects instead of the programmatic interface to the data dictionary. See Optimize, Estimate, and Sweep Block Parameter Values.

You must have a Parallel Computing Toolbox™ license to perform parallel simulation.

% For convenience, define names of model and data dictionary
model = 'mySweepMdl';
dd = 'mySweepDD.sldd';

% Define the sweeping values for the variant control
CtrlValues = [1 2 3 4];

% Grant each worker in the parallel pool an independent data dictionary 
% so they can use the data without interference
spmd 
    Simulink.data.dictionary.setupWorkerCache
end

% Determine the number of times to simulate
numberOfSims = length(CtrlValues);

% Prepare a nondistributed array to contain simulation output
simOut = cell(1,numberOfSims);

parfor index = 1:numberOfSims
    % Create objects to interact with dictionary data
    % You must create these objects for every iteration of the parfor-loop
    dictObj = Simulink.data.dictionary.open(dd);
    sectObj = getSection(dictObj,'Design Data');
    entryObj = getEntry(sectObj,'MODE'); 
    % Suppose MODE is a Simulink.Parameter object stored in the data dictionary
    
    % Modify the value of MODE
    temp = getValue(entryObj);
    temp.Value = CtrlValues(index);
    setValue(entryObj,temp);

    % Simulate and store simulation output in the nondistributed array
    simOut{index} = sim(model);
    
    % Each worker must discard all changes to the data dictionary and
    % close the dictionary when finished with an iteration of the parfor-loop
    discardChanges(dictObj);
    close(dictObj);
end

% Restore default settings that were changed by the function
% Simulink.data.dictionary.setupWorkerCache
% Prior to calling cleanupWorkerCache, close the model

spmd
    bdclose(model)
    Simulink.data.dictionary.cleanupWorkerCache
end

Note

If data dictionaries are open, you cannot use the command Simulink.data.dictionary.cleanupWorkerCache. To identify open data dictionaries, use Simulink.data.dictionary.getOpenDictionaryPaths.

Data Concurrency Issues

Data concurrency issues refer to scenarios for which software makes simultaneous attempts to access the same file for data input or output. In Simulink, they primarily occur as a result of the nonsequential nature of the parfor loop during simultaneous execution of Simulink models. The most common incidences arise when code is generated or updated for a simulation target of a Stateflow^®, Model block or MATLAB Function block during parallel computing. The cause, in this case, is that Simulink tries to concurrently access target data from multiple worker sessions. Similarly, To File blocks may simultaneously attempt to log data to the same files during parallel simulations and thus cause I/O errors. Or a third-party blockset or user-written S-function may cause a data concurrency issue while simultaneously generating code or files.

A secondary cause of data concurrency is due to the unprotected access of network ports. This type of error occurs, for example, when a Simulink product provides blocks that communicate via TCP/IP with other applications during simulation. One such product is the HDL Verifier™ for use with the Mentor Graphics^® ModelSim^® HDL simulator.

Resolving Data Concurrency Issues

The core requirement of parfor is the independence of the different iterations of the parfor body. This restriction is not compatible with the core requirement of simulation via incremental code generation, for which the simulation target from a prior simulation is reused or updated for the current simulation. Hence during the parallel simulation of a model that involves code generation (such as Accelerator mode simulation), Simulink makes concurrent attempts to access (update) the simulation target. However, you can avoid such data concurrency issues by creating a temporary folder within the parfor loop and then adding several lines of MATLAB code to the loop to perform the following steps:

1. Change the current folder to the temporary, writable folder.

2. In the temporary folder, load the model, set parameters and input vectors, and simulate the model.

3. Return to the original, current folder.

4. Remove the temporary folder and temporary path.

In this manner, you avoid concurrency issues by loading and simulating the model within a separate temporary folder. Following are examples that use this method to resolve common concurrency issues.

A Model with Stateflow, MATLAB Function Block, or Model Block.  In this example, either the model is configured to simulate in Accelerator mode or it contains a Stateflow, a MATLAB Function block, or a Model block (for example, sf_bounce or sldemo_autotrans). For these cases, Simulink generates code during the initialization phase of simulation. Simulating such a model in parfor would cause code to be generated to the same files, while the initialization phase is running on the worker sessions. As illustrated below, you can avoid such data concurrency issues by running each iteration of the parfor body in a different temporary folder.

parfor i=1:4
   cwd = pwd;
   addpath(cwd)
   tmpdir = tempname;
   mkdir(tmpdir)
   cd(tmpdir)
   load_system(model)
   % set the block parameters, e.g., filename of To File block
   set_param(someBlkInMdl, blkParamName, blkParamValue{i})
   % set the model parameters by passing them to the sim command
   out{i} = sim(model, mdlParamName, mdlParamValue{i});
   close_system(model,0);
   cd(cwd)
   rmdir(tmpdir,'s')
   rmpath(cwd)
end

Note the following:

• You can also avoid other concurrency issues due to file I/O errors by using a temporary folder for each iteration of the parfor body.

• On Windows^® platforms, consider inserting the evalin('base', 'clear mex'); command before rmdir(tmpdir, 's'). This sequence closes MEX-files first before calling rmdir to remove tmpdir.

evalin('base', 'clear mex');
rmdir(tmpdir, 's')

A Model with To File Blocks.  If you simulate a model with To File blocks from inside of a parfor loop, the nonsequential nature of the loop may cause file I/O errors. To avoid such errors during parallel simulations, you can either use the temporary folder idea above or use the sim command in Rapid Accelerator mode with the option to append a suffix to the file names specified in the model To File blocks. By providing a unique suffix for each iteration of the parfor body, you can avoid the concurrency issue.

rtp = Simulink.BlockDiagram.buildRapidAcceleratorTarget(model); 
       parfor idx=1:4 
       sim(model, ... 
           'ConcurrencyResolvingToFileSuffix', num2str(idx),... 
           'SimulationMode', 'rapid',... 
           'RapidAcceleratorUpToDateCheck', 'off'); 
        end

Related Topics

• Model an Automatic Transmission Controller
• Optimize, Estimate, and Sweep Block Parameter Values
• Sweep Variant Control Using Parallel Simulation