distributed

Create and access elements of distributed arrays from client

Description

A distributed array on the client represents an array that is partitioned out among the workers in a parallel pool. You operate on the entire array as a single entity; however, workers operate only on their part of the array and automatically transfer data between themselves when necessary. A distributed array resembles a normal MATLAB^® array in the way you index and manipulate its elements, but none of its elements exist on the client. Codistributed arrays that you create inside spmd statements are accessible as distributed arrays from the client.

Creation

Use the distributed function or use the "distributed" option of array creation functions such as ones or zeros. For a list of array creation functions that create distributed arrays directly on workers, see Alternative Functionality.

Syntax

D = distributed(ds)

D = distributed(X)

D = distributed(C,dim)

D = distributed(tX)

Description

D = distributed(ds) creates a distributed array from a datastore ds. D is a distributed array stored in parts on the workers of the open parallel pool.

To retrieve the distributed array elements from the pool back to an array in the MATLAB workspace, use the gather function.

D = distributed(X) creates a distributed array from an array X.

Use this syntax to create a distributed array from local data only if the MATLAB client can store all of X in memory. To create large distributed arrays, use the previous syntax to create a distributed array from a datastore, or the "distributed"option of array creation functions such as ones, zeros, or any other creation functions listed in Alternative Functionality.

If the input argument is already a distributed array, the result is the same as the input.

D = distributed(C,dim) creates a distributed array from the Composite object C, with the entries of C concatenated and distributed along the dimension dim. If you omit dim, then the first dimension is the distribution dimension.

All entries of the Composite object must have the same class. Dimensions other than the distribution dimension must be the same.

D = distributed(tX) converts the tall array tX into a distributed array distributed along the first dimension. tX must be defined in a parallel environment that can run distributed arrays. (since R2023b)

Input Arguments

ds — Datastore

TabularTextDatastore object | ImageDatastore object | SpreadsheetDatastore object | KeyValueDatastore object | FileDatastore object | TallDatastore object | ...

Datastore, specified as one of the following objects.

Object	Type
TabularTextDatastore object	Text files
ImageDatastore object	Image files
SpreadsheetDatastore object	Spreadsheet files
KeyValueDatastore object	MAT files as well as sequence files you produce using mapreduce
FileDatastore object	Custom format files
TallDatastore object	MAT-files and sequence files produced by the write function of the tall data type
ParquetDatastore object	Parquet files
DatabaseDatastore (Database Toolbox) object	Database

X — Array to distribute

array

Array to distribute, specified as an array.

C — Composite object to distribute

Composite object

Composite object to distribute, specified as a Composite object.

dim — Distribution dimension

scalar integer

Distribution dimension, specified as a scalar integer. The distribution dimension specifies the dimension over which you distribute the Composite object.

tX — Tall array to convert

tall array

Since R2023b

Tall array to convert to a distributed array, specified as a tall array. The tall array must be defined in a parallel environment that supports distributed arrays.

Output Arguments

D — Distributed array

distributed array

Distributed array stored in parts on the workers of the open parallel pool, returned as a distributed array.

Object Functions

gather	Transfer distributed array, Composite object, or gpuArray object to local workspace
write	Write distributed data to an output location

Several MATLAB toolboxes include functions with distributed array support. For a list of functions in all MathWorks^® products that support distributed arrays, see All Functions List (Distributed Arrays).

Several object functions enable you to examine the characteristics of a distributed array. Most behave like the MATLAB functions of the same name.

isdistributed	True for distributed array
isreal	Determine whether array uses complex storage
isUnderlyingType	Determine whether input has specified underlying data type
length	Length of largest array dimension
ndims	Number of array dimensions
size	Array size
underlyingType	Type of underlying data determining array behavior

Examples

Create Distributed Arrays from Datastores

This example shows how to create and load distributed arrays using datastore.

First, create a datastore using an example data set. This data set is too small to show equal partitioning of the data over the workers. To simulate a large data set, artificially increase the size of the datastore using repmat.

files = repmat("airlinesmall.csv",10,1);
ds = tabularTextDatastore(files);

Select the example variables.

ds.SelectedVariableNames = ["DepTime", "DepDelay"];
ds.TreatAsMissing = "NA";

Create a distributed table by reading the datastore in parallel. Partition the datastore with one partition per worker. Each worker then reads all data from the corresponding partition. The files must be in a shared location accessible from the workers.

dt = distributed(ds);

Starting parallel pool (parpool) using the 'Processes' profile ... connected to 4 workers.

Finally, display summary information about the distributed table.

summary(dt) 

Variables:

    DepTime: 1,235,230×1 double
        Values:

            min          1
            max       2505
            NaNs    23,510

    DepDelay: 1,235,230×1 double
        Values:

            min      -1036
            max       1438
            NaNs    23,510

Create Distributed Arrays

This example shows how to create and retrieve distributed arrays.

Create a small array and convert it into a distributed array.

Nsmall = 50;
D1 = distributed(magic(Nsmall));

Starting parallel pool (parpool) using the 'Processes' profile ...
Connected to parallel pool with 4 workers.

Create a large distributed array directly on the workers by using a build function.

Nlarge = 1000;
D2 = rand(Nlarge,"distributed");

Retrieve elements of a distributed array back to the local workspace. You can use whos to determine the location of the data in the workspace by examining the Class variable.

D3 = gather(D2);
whos

  Name           Size                Bytes  Class          Attributes

  D1            50x50                20000  distributed              
  D2          1000x1000            8000000  distributed              
  D3          1000x1000            8000000  double                   
  Nlarge         1x1                     8  double                   
  Nsmall         1x1                     8  double                   

Create Distributed Array from Composite Object

Start a parallel pool of workers and create a Composite object by using spmd.

p = parpool("Processes",4);

Starting parallel pool (parpool) using the 'Processes' profile ...
Connected to parallel pool with 4 workers.

spmd
    C = rand(3,spmdIndex-1);
end
C

 
C =
 
   Worker 1: class = double, size = [3  0]
   Worker 2: class = double, size = [3  1]
   Worker 3: class = double, size = [3  2]
   Worker 4: class = double, size = [3  3]
 

To create a distributed array from the Composite object, use the distributed function. For this example, distribute the entries along the second dimension.

d = distributed(C,2)

d =

    0.6383    0.9730    0.2934    0.3241    0.9401    0.1897
    0.5195    0.7104    0.1558    0.0078    0.3231    0.3685
    0.1398    0.3614    0.3421    0.9383    0.3569    0.5250

Examine how the distribution of the data on the workers.

spmd
    d
end

Worker 1: 
  This worker does not store any elements of d.
Worker 2: 
  This worker stores d(:,1).
          LocalPart: [3x1 double]
      Codistributor: [1x1 codistributor1d]
Worker 3: 
  This worker stores d(:,2:3).
          LocalPart: [3x2 double]
      Codistributor: [1x1 codistributor1d]
Worker 4: 
  This worker stores d(:,4:6).
          LocalPart: [3x3 double]
      Codistributor: [1x1 codistributor1d]

When you are finished with the computations, delete the parallel pool.

delete(p);

Parallel pool using the 'Processes' profile is shutting down.

Convert Tall Arrays to Distributed Arrays

Since R2023b

This example shows how to convert a tall array into a distributed array.

Create a tall table using an example data set. If you have Parallel Computing Toolbox™ installed, when you use the tall function, MATLAB automatically starts a parallel pool of workers unless you turn off the default parallel pool preference. The default cluster uses local process workers on your machine.

size = 2000000;
tt = tall(table((1:size)',randn(size,1),randn(size,1),randn(size,1), ...
    'VariableNames',["Exp","Rep1","Rep2","Rep3"]))

Starting parallel pool (parpool) using the 'Processes' profile ...
Connected to parallel pool with 4 workers.

tt =

  2,000,000×4 tall table

    Exp      Rep1        Rep2         Rep3  
    ___    ________    _________    ________

     1      -1.6003      -1.2403     -1.6124
     2      0.76827      0.33907     0.77811
     3       1.4637      0.74255    -0.91635
     4      -1.2478      0.10478    -0.10097
     5        0.619      -1.2974     0.93445
     6       1.8375       0.2142     -1.1036
     7     -0.44354      -1.1438    -0.89011
     8      -1.3351    -0.036768     0.65196
     :        :            :           :
     :        :            :           :

Convert the tall table into a distributed table. MATLAB partitions the data in the tall table along the first dimension and distributes it to the workers.

dt = distributed(tt);

Display summary information about the distributed table.

summary(dt)

Variables:

    Exp: 2,000,000×1 double
        Values:

            Min         1  
            Max     2e+06  

    Rep1: 2,000,000×1 double
        Values:

            Min    -5.1402 
            Max     4.8763 

    Rep2: 2,000,000×1 double
        Values:

            Min    -4.7961 
            Max     4.9875 

    Rep3: 2,000,000×1 double
        Values:

            Min    -4.8369 
            Max     5.1454 

Finally, examine how much data is stored on each worker. The data is partitioned evenly over the workers.

spmd
    dt
end

Worker 1: 
  
  This worker stores dt(1:500000,:).
  
          LocalPart: [500000x4 table]
      Codistributor: [1x1 codistributor1d]
  
Worker 2: 
  
  This worker stores dt(500001:1000000,:).
  
          LocalPart: [500000x4 table]
      Codistributor: [1x1 codistributor1d]
  
Worker 3: 
  
  This worker stores dt(1000001:1500000,:).
  
          LocalPart: [500000x4 table]
      Codistributor: [1x1 codistributor1d]
  
Worker 4: 
  
  This worker stores dt(1500001:2000000,:).
  
          LocalPart: [500000x4 table]
      Codistributor: [1x1 codistributor1d]
  

Tips

• A distributed array is created on the workers of the existing parallel pool. If no pool exists, distributed starts a new parallel pool unless the automatic starting of pools is disabled in your parallel preferences. If there is no parallel pool and distributed cannot start one, MATLAB returns the result as a nondistributed array in the client workspace.

Alternative Functionality

This table lists the available MATLAB functions that create distributed arrays directly on the workers. For more information, see the Extended Capabilities section of the function reference page.

eye	distributed.cell
false	distributed.colon
Inf	distributed.linspace
NaN	distributed.logspace
ones	distributed.spalloc
true	distributed.speye
zeros	distributed.sprand
rand	distributed.sprandn
randi
randn

Version History

Introduced in R2008a

R2023b: Convert tall arrays to distributed arrays

Use the distributed function to convert a tall array to a distributed array and access MATLAB functions that have distributed array support.

In previous releases, when you used a tall array with the distributed function, MATLAB threw an error.