How to convert the format of data from sequences to matrices when designing deep learning networks?

Hello,

After importing the network into DeepNetworkDesigner for analysis, I encountered the following problem: after being processed by selfattentionLayer, the data size format is 577 (S) x 577 (C) x 1 (B).

I want to convert it to a format similar to imageInputLayer, _ (S) x_ (S) x_ (C) x_ (B). How can I use MATLAB to implement it?

The code for the network is as follows:

patchSize = 16;
embeddingOutputSize = 768;
layer = patchEmbeddingLayer(patchSize,embeddingOutputSize)

net = dlnetwork;

inputSize = [384 384 3];

maxPosition = (inputSize(1)/patchSize)^2 + 1;

numHeads = 4;
numKeyChannels = 4*embeddingOutputSize;

numClasses = 1000;

layers = [ 
    imageInputLayer(inputSize)
    patchEmbeddingLayer(patchSize,embeddingOutputSize,Name="patch-emb")
    embeddingConcatenationLayer(Name="emb-cat")
    positionEmbeddingLayer(embeddingOutputSize,maxPosition,Name="pos-emb");
    additionLayer(2,Name="add")
    selfAttentionLayer(numHeads,numKeyChannels,AttentionMask="causal",OutputSize=maxPosition)
    fullyConnectedLayer(numClasses)
    softmaxLayer];
net = addLayers(net,layers);

net = connectLayers(net,"emb-cat","add/in2");

[Cris LaPierre]

Are you sure your data is 577 (S) x 577 (C) x 1 (B)?

Could it instead be 577 (S) x 577 (S) x 1 (C) or 577 (S) x 577 (S) x 1 (B)?

[Tian,HCong]

Thank you very much for your prompt reply.

The corresponding data is indeed 577 (S) x 577 (C) x 1 (B), as shown in the data graph for each layer.

[Cris LaPierre]

Can you share a sample data set? That will make it easier to help.

[Tian,HCong]

@Cris LaPierre

Thank you very much for your prompt reply.

This dataset comes from the folder in Matlab and is commonly used by me when testing simple deep learning networks.

Answer 1

[Gayathri]

Hello @Tian,HCong,

I understand that you want to convert a ‘SCB’ formatted output to ‘SSCB’ formatted output. It can be achieved as shown below.

X = dlarray(rand(577,577),'SCB');
out = X(2:end, :, :);
WH = sqrt(size(out, 1)); % Calculate the spatial dimensions
C = size(out, 2); % Number of channels
D=size(out,3);
Z = reshape(out, WH, WH, C, D); 
Z=dlarray(Z,'SSCB')

For more information about “dlarray”, please refer to the following MathWorks documentation link.

https://www.mathworks.com/help/releases/R2024a/deeplearning/ref/dlarray.html

Hope you find this information helpful.