How to make a regression network from a semantic segmentation network?

Hello , I have just used a semantic segmentation network to detect patterns from a copper deposit but i want to know if I can get a regression network out of this . I mean the semantic segmentation network input is a 744x1432x3 image size with a output 744x1432x7 size (7 because there is 7 labels) , but the point is recognising the output state of the image.The code i get so far below:

load '/MATLAB Drive/RLE/G_truth/gTruth.mat'
[imds,pxdsTruth] = pixelLabelTrainingData(gTruth)
dsTrain = combine(imds,pxdsTruth);
trainingData_2 = combine(imds,pxdsTruth);
tbl = countEachLabel(pxdsTruth)
frequency = tbl.PixelCount/sum(tbl.PixelCount);
classes = ["Quemado","Mate","Estriado","Aspero","Blanquecino","Deposito","Fondo"];
numClasses = numel(classes);

% Create DeepLab v3+.
lgraph = deeplabv3plusLayers(imageSize, numClasses, "resnet18");
options = trainingOptions('sgdm', ...
    'LearnRateSchedule','piecewise',...
    'LearnRateDropPeriod',12,...
    'LearnRateDropFactor',0.3,...
    'Momentum',0.9, ...
    'InitialLearnRate',1e-3, ...
    'L2Regularization',0.005,...
    'MaxEpochs',14, ...  
    'MiniBatchSize',2, ...
    'Shuffle','every-epoch', ...
    'CheckpointPath', tempdir, ...
    'VerboseFrequency',2,...
    'Plots','training-progress',...
    'ValidationPatience', 4);
[net] = trainNetwork(dsTrain,lgraph,options);

from this input

I get an image like this ,as instance

But this is an output from a reagents blending that undergoes a electrodeposition process , so I have 80 images that means there are 80 blendings each (a table with the composition each blending). My question is if I need to make a second algorithm network for regression (with input the labeled image) and output(the predicted blending composition) apart from the semantic or If possible can fuse them into one, because I am aiming to get the blending composition from the original image. Thanks for the help.

I attach the network I use: resnet18

Answer 1

[Avadhoot]

Hi @Raúl Rivera,

Your problem seems to be comprised of 2 distinct tasks i.e. segmentation and regression. This problem could be solved with 2 approaches. I will list both the approaches below along with their pros and cons.

1) Sequential Approach:

As you have already constructed and trained a segmentation network in the form of a "resnet18" network it would be convenient to follow a sequential approach and construct another smaller regression network after "resnet18" to predict the blending composition. The regression task would require the following steps:

1. Feature extraction: Extract meaningful features from the segmented images to feed into the regression network. Features could be extracted by summarizing the segmented areas, taking into account the texture and patterns or simply using the raw pixel data of certain segments.
2. Regression model: Train and test a regression model which takes in the extracted features as an input and predicts the blending composition. As you already have the data for the blending composition, the training and testing of the network will be easy.

Pros:

1. Modular approach: This approach consists of 2 distinct modules performing 2 distinct tasks. This is a highly desirable trait.
2. Easier to debug: The problems encountered in this approach can be easily traced back to the point of origin due to the modular structure of the solution.
3. Freedom to use different models: You are free to use different models for different tasks as they have no dependency on each other except the input image.

Cons:

1. Requires feature extraction to be done manually before the regression phase.
2. Does not provide end-to-end solution.

2) Integrated approach:

This is another approach you can use to achieve the same results. Here you need to modify the neural network architecture such that the regression step also gets performed in the network itself. You can add a parallel branch to the existing architecture or you can modify the layers in such a way that regression predictions can also be done.

You need a custom loss function which incorporates loss for both segmentation and regression tasks. The training data also needs to be modified so that it contains the segmentation labels and the blending composition outputs.

Pros:

1. End to end approach which gives blending composition in the end.
2. Integrated feature extracting into the network.
3. Could potentially perform better.

Cons:

1. Complex network architecture.
2. Difficult to train due to the custom loss function and modifications to the data.
3. Requires in-depth knowledge about the network architecture to ensure balance between both the tasks.
4. Difficult to understand and debug.

In summary, it would be beneficial to use the sequential approach if you desire modularity, simplicity and a straightforward approach. You can prefer the integrated approach if you are comfortable to deal with the complexity of the model.

I hope this helps.