Issue implementing custom fully connected layer - weights converging to 0
12 views (last 30 days)
I'd like to implement a custimized version of the fully connected layer. Before that, I'd like to make sure I can replicate the default FullyConnectedLayer that comes with Matlab. While my implementation did not return errors, the weights the network learned were unusually small. In my simulation, the default FullyConnectedLayer returned weights mostly between [1,0]. My implementation returned weights around 0 (e.g., 10^-40). Can someone help explain why this is happening?
classdef CustomFCLayer < nnet.layer.Layer
% Custom fully connected layer, constrain weights to be positive.
% Layer learnable parameters
function layer = CustomFCLayer(prev_units,num_units,name,initWeights)
% layer = CustomFCLayer(numInputs,name) creates a
% fully connected layer
layer.Name = name;
layer.Description = 'Fully connected layer';
std = sqrt(2/(num_units+prev_units));
layer.Weights = std*rand([num_units prev_units]);
layer.Weights = initWeights;
layer.Bias = zeros([num_units 1]);
function Z = predict(layer, X)
% Forward input data through the layer at prediction time and
% output the result
Z = (layer.Weights)*X+layer.Bias;
Anshika Chaurasia on 11 Feb 2021
As @Larry mentioned by default the weights are initialized with Glorot initializer in fully connected layers provided by MATLAB.
Refer to following link for more information about weightsInitializer in fully connected layer:
Refer to following documentation for writing custom initializeGlorot function:
Hope it helps!