More detailed documentation on Deep Learning Toolbox normalization layers?

Below are excerpts from the Deep Learning Toolbox documentation describing several neural network layer types that perform different kinds of input data normalization. For me, the decriptions are a bit too terse and textual to understand clearly the differences between the normalization operations that these different layers apply (both at training time and at test time). Is there any additional documentation to be found somewhere, with a more mathematical description, and perhaps illustrative figures?

imageInputLayer An image input layer inputs 2-D images to a network and applies data normalization.

batchNormalizationLayer A batch normalization layer normalizes a mini-batch of data across all observations for each channel independently. To speed up training of the convolutional neural network and reduce the sensitivity to network initialization, use batch normalization layers between convolutional layers and nonlinearities, such as ReLU layers.

groupNormalizationLayer A group normalization layer normalizes a mini-batch of data across grouped subsets of channels for each observation independently. To speed up training of the convolutional neural network and reduce the sensitivity to network initialization, use group normalization layers between convolutional layers and nonlinearities, such as ReLU layers.

instanceNormalizationLayer An instance normalization layer normalizes a mini-batch of data across each channel for each observation independently. To improve the convergence of training the convolutional neural network and reduce the sensitivity to network hyperparameters, use instance normalization layers between convolutional layers and nonlinearities, such as ReLU layers.

layerNormalizationLayer A layer normalization layer normalizes a mini-batch of data across all channels for each observation independently. To speed up training of recurrent and multilayer perceptron neural networks and reduce the sensitivity to network initialization, use layer normalization layers after the learnable layers, such as LSTM and fully connected layers.

Answer 1

[Tish Sheridan]

Hi! Did you find the Algorithms sections at the end? (on each of those layer pages except the input layer, eg https://www.mathworks.com/help/deeplearning/ref/nnet.cnn.layer.batchnormalizationlayer.html#d123e18507)

Any help?

[Matt J]

It is of some help, Tish, thanks. However, I find it vague in a few respects. For instance, for the groupNormalizationLayer doc where it says,

"normalizes the elements of the input by first calculating the mean and variance over spatial, time, and grouped subsets of the channel dimensions for each observation independently."

Isn't a vector here representing the i-th observation and doesn't "time" refer to the variation of over i, that is from observation to observation? If the averaging is being done over i, how can it then be that and are calculated "for each observation independently"?

Also, what happens at inference time? Multiple different and have been calculated from different sections of the training data, and would need to be consolidated in some way. Are they just averaged together?

[Ieuan Evans]

For simplicity, here refer the elements of the input data X where X is an N-D array. The dimensionality of the input depends on the type of data (e.g. 2-D images, 3-D images, sequences, etc.). For example, in these algorithm descriptions, can denote a single channel of a pixel (e.g. the R value of a pixel an RGB image).

Here, and in most deep learning data contexts, the term "time" refers to temporal dimension of sequence data. For example, if the data is an numChannels-by-numObservations-by-numTimeSteps array representing a batch of sequences, then the time dimension is the third dimension.

For example, if you have video data represented as a H-by-W-by-C-by-numObservations-by-numTimeSteps array, you can normalize over the spatial dimensions (1 and 2), the channel dimension (3), and the time dimension (5) independently of the observation dimension (4).

Group, instance, and layer normalization layers normalize mini-batches independently and calculate a fresh μ and for each mini-batch. They behave the same in training and inference time. Batch normalization layers behave differently. They use the calculated mini-batch statistics at training time, but use the aggregated μ and calculated for the training data for inference.

Figure 2 of this paper has some handy diagrams of different types of normalization layers:

https://arxiv.org/pdf/1803.08494.pdf

[Matt J]

Excellent, thanks!