How to train CNN with an image in the input and an image in the output?

I have a task to train CNN with an image as input and an image as output. I have tried to do it at the beginning with Matlab tutorial, but matlab has no image as output, but a vector.

https://de.mathworks.com/help/deeplearning/examples/train-a-convolutional-neural-network-for-regression.html

any idea how i can do this tutorial again but with image as output?

For example you could insert input as image for number one and the output is also for number one but rotated or deformed.

Please don't send me other people's question, because i have seen all answers about it.

[ghada sandoub]

Hello Osama,

I hope that you have found a solution to your problem. I have the same situation, I need to train CNN for image enhancement and I don't know how to design the network to recieve an image as input and the output is also an image not a label. However, all the available CNNs in matlab can recieve an image as input and the output is a label for the task of classification or regression. could you please if you already found a method to do this please help me on that. Thanks alot.

[Osama Tabbakh]

Hello Ghada,

I gave up after almost 2 months working on that! I used instead of a normal neural network with the algorithm Levenberg Marquardt.
But for that you need to describe your output in another way.

I think it is possible but not in Matlab! I am working right now with python, that offers more flexibility especially in structural the CNN layers. 

When you like, I can tell you more about my experience. You should just write me.
And when I know how to do this in python, I will let you know. 

I wish you all the best with your project, good luck

Sincerely yours,
Osama

[ghada sandoub]

Thanks alot Osama for your reply. However, unfortunately, I have no experience in using python. So, I was searching for a method that can do this on Matlab.

[Ritesh Jain]

Hello Osama,

I am stuck in the same thing for almost a month can you please help me out in this thing in Python.

[Osama Tabbakh]

Hello Ritesh,

Unfortunately I haven't done it in Python yet. But I wish you all the best with it, good luck.

[Florent Busnoult]

There is an example in the 5G toolbox that uses a CNN network with a picture as an input and a picture as an output.

https://www.mathworks.com/help/5g/examples/deep-learning-data-synthesis-for-5g-channel-estimation.html

The variable used to store the "labels" or (ouput images) is the following:

trainLabels(:,:,:,i) = est_grid;

It's a 4-D double variable.

And you can look at the different output images used to trian the network by using the command below:

>> imagesc(abs(trainLabels(:,:,:,2))); % output image number 2

>> imagesc(abs(trainLabels(:,:,:,3))); % output image number 3

>> imagesc(abs(trainLabels(:,:,:,25))); % output image number 25

and so on.

Answer 1

[Vasilis Giannoglou]

I was also struggling with having an image as an output and I found the solution. Skip the fullyConnectedLayer(n) part, because it causes the network to have an output of either a vector or just a number. However, make sure not to include any Pooling layers if you want your output to be of the same size as your input, because they cause downsampling. Finally, keep in mind that the number of filters of your final conv network is the same number as the 3rd dimension of your output image. For example,

convolution2dLayer(3,1,'Padding','same') gives an output image of (M,N,1), where M,N are the rows and columns of the initial image. The next code runs to me.

layers = [
    imageInputLayer([64 64 1])  % My initial image is 64x64

    convolution2dLayer(3,8,'Padding','same')
    batchNormalizationLayer
    reluLayer
    
%     averagePooling2dLayer(2,'Stride',1)  % Don't want this

    convolution2dLayer(3,16,'Padding','same')
    batchNormalizationLayer
    reluLayer
    
%     averagePooling2dLayer(2,'Stride',1) % Don't want this
  
    convolution2dLayer(3,32,'Padding','same')
    batchNormalizationLayer
    reluLayer
    
    convolution2dLayer(3,1,'Padding','same') % Because my initial image 3rd dimension is one.
    batchNormalizationLayer
    reluLayer
      
    dropoutLayer(0.2)
%     fullyConnectedLayer(10) % Don't want this
    regressionLayer];

[Osama Tabbakh]

it really worked out, I think this tutorial is the right one:

https://www.mathworks.com/help/5g/examples/deep-learning-data-synthesis-for-5g-channel-estimation.html

Thanks Vasilis Giannoglou Florent Busnoult.

[mary john]

I tried to do this and gets an error with the regression layer. It states that the ''output size does not match response size", when I use the deep network designer.

Can you please help me to solve. Thanks in advance

Answer 2

[Johanna Pingel]

Try this example: https://www.mathworks.com/help/deeplearning/examples/image-to-image-regression-using-deep-learning.html

Let me know if this doesn't answer your question!

[Osama Tabbakh]

I've seen this before and I don't know if I can apply this on my project. I have a picture for the magnet field as Input and coil as output. Should I use autoencoder just like this tutorial? For me, this tutorial isn't clear.

[supriya Naik]

Did you get the solution sir?? Actually I am facing same problem now.

[Osama Tabbakh]

Yes, please see the accepted answer