Except for the first iteration others show NaN

When I start training my CNN it shows me training details only on first iteration, and I've already check the training data size and the validation data size is 100000 and 20000, I was wondering is it normal process or not. Thank you.

[Chunru]

Check if the train data contains any nan.

[KAI-YANG WANG]

Thank you for the answer, Ive checked there's no nan in my training data.

Answer 1

[KSSV]

Try reducing the initial learning rate.

options = trainingOptions('adam') ;
options.InitialLearnRate

ans = 1.0000e-03

options.InitialLearnRate = 10^-4 ;
options.InitialLearnRate

ans = 1.0000e-04

If the above works well and good. If not show us your full code and data.

[KAI-YANG WANG]

Thank you so much and I've try to reduce the initial learning rate, but still the same. My code is attached below.

location='\\ikb\home\25\1939725\Desktop\cost2100-master mimo\matlab\CNN\csitrain\Htrain'
imds = imageDatastore(location,'FileExtensions','.mat','IncludeSubfolders',1, ...
                      'LabelSource','foldernames');
imdsx=imageDatastore(location,'FileExtensions','.mat','IncludeSubfolders',1, ...
                      'LabelSource','foldernames');
imdsy=imageDatastore(location,'FileExtensions','.mat','IncludeSubfolders',1, ...
                      'LabelSource','foldernames');
                  
countEachLabel(imds)
trainingDS = imds;
trainingDS.Labels = categorical(trainingDS.Labels);
trainingDS.ReadFcn=@readFcn1;

countEachLabel(imdsx)
trainingDS2 = imdsx;
trainingDS2.Labels = categorical(trainingDS.Labels);
trainingDS2.ReadFcn=@readFcn2;

countEachLabel(imdsy)
trainingDS2 = imdsy;
trainingDS2.Labels = categorical(trainingDS.Labels);
trainingDS2.ReadFcn=@readFcn3;


dsTrain = transform(imds,@commonPreprocessing);
dsVal = transform(imdsx,@commonPreprocessing);
dstest = transform(imdsy,@commonPreprocessing);

trainData=combine(dsTrain,dsTrain);
valData=combine(dsVal,dsVal);


%Designed concolutional network


maxEpochs = 200;
options = trainingOptions('sgdm', ...
    'ValidationData', valData,...
    'InitialLearnRate',0.0001,...
    'ExecutionEnvironment','gpu', ...
    'LearnRateSchedule','piecewise', ...
    'MaxEpochs',maxEpochs, ...
    'MiniBatchSize',200, ...
    'Plots','training-progress')

doTraining = true; 
if doTraining  
    modelDateTime = string(datetime('now','Format',"yyyy-MM-dd-HH-mm-ss"));
    net =trainNetwork(trainData, lgraph, options);
    save(strcat("trainedCNN-",modelDateTime,"-Epoch-",num2str(maxEpochs),".mat"),'net');
else 
    load('traindata.mat'); 
end

YPredicted = predict(net,dstest);

function dataOut = commonPreprocessing(data)

    dataOut = cell(size(data));
    for col = 1:size(data,2)
        for idx = 1:size(data,1)
            temp = single(data{idx,col});
            temp = reshape(temp,[32,32,2]);
            temp = rescale(temp);
            dataOut{idx,col} = temp;
        end
    end
end

[KAI-YANG WANG]

I've tried a further lower learning rate and it does help, Thank you!