Issue implementing custom fully connected layer - weights converging to 0

I'd like to implement a custimized version of the fully connected layer. Before that, I'd like to make sure I can replicate the default FullyConnectedLayer that comes with Matlab. While my implementation did not return errors, the weights the network learned were unusually small. In my simulation, the default FullyConnectedLayer returned weights mostly between [1,0]. My implementation returned weights around 0 (e.g., 10^-40). Can someone help explain why this is happening?

classdef CustomFCLayer < nnet.layer.Layer
    % Custom fully connected layer, constrain weights to be positive.

    properties (Learnable)
        % Layer learnable parameters
        Weights
        Bias
    end
    
    methods
        function layer = CustomFCLayer(prev_units,num_units,name,initWeights) 
            % layer = CustomFCLayer(numInputs,name) creates a
            % fully connected layer 

            layer.Name = name;
            layer.Description = 'Fully connected layer';
            
            if ~exist('initWeights','var')
                std = sqrt(2/(num_units+prev_units));
                layer.Weights = std*rand([num_units prev_units]);
            else
                assert(size(initWeights,1)==num_units) 
                assert(size(initWeights,2)==prev_units) 
                layer.Weights = initWeights;
            end
            
            layer.Bias = zeros([num_units 1]);
        end
        
        function Z = predict(layer, X)
            % Forward input data through the layer at prediction time and
            % output the result
            Z = (layer.Weights)*X+layer.Bias;
        end
    end
end

[Larry Llewellyn]

I noticed at the following link the MATLAB provied fullyConnectedLayer uses 'glorot' to initialize the weights with the Glorot initializer:

https://www.mathworks.com/help/deeplearning/ref/nnet.cnn.layer.fullyconnectedlayer.html#mw_bad3166a-93e7-42c0-8bd2-740cd2a842ad

Answer 1

[Anshika Chaurasia]

Hi Kenny,

As @Larry mentioned by default the weights are initialized with Glorot initializer in fully connected layers provided by MATLAB.

Refer to following link for more information about weightsInitializer in fully connected layer:

https://www.mathworks.com/help/deeplearning/ref/nnet.cnn.layer.fullyconnectedlayer.html#mw_ba098309-7bb2-4c1b-ad87-1246691c3407

Refer to following documentation for writing custom initializeGlorot function:

https://www.mathworks.com/help/deeplearning/ug/initialize-learnable-parameters-for-custom-training-loop.html?searchHighlight=glorot%20initialization&s_tid=srchtitle#mw_1bd0f2c3-c7df-4841-89ce-a7574d2db8d9

Hope it helps!