Main Content

Group normalization layer

A group normalization layer normalizes a mini-batch of data across grouped subsets of channels for each observation independently. To speed up training of the convolutional neural network and reduce the sensitivity to network initialization, use group normalization layers between convolutional layers and nonlinearities, such as ReLU layers.

After normalization, the layer scales the input with a learnable scale factor
*γ* and shifts it by a learnable offset
*β*.

creates a group normalization layer and sets the optional `layer`

= groupNormalizationLayer(`numGroups`

,`Name,Value`

)`'Epsilon'`

, Parameters and Initialization, Learning Rate and Regularization, and `Name`

properties using one or more name-value arguments. You can
specify multiple name-value arguments. Enclose each property name in quotes.

The group normalization operation normalizes the elements
*x _{i}* of the input by first calculating the mean

$${\widehat{x}}_{i}=\frac{{x}_{i}-{\mu}_{G}}{\sqrt{{\sigma}_{G}^{2}+\epsilon}},$$

where *ϵ* is a constant that improves numerical
stability when the variance is very small. To allow for the possibility that inputs with
zero mean and unit variance are not optimal for the operations that follow group
normalization, the group normalization operation further shifts and scales the activations
using the transformation

$${y}_{i}=\gamma {\widehat{x}}_{i}+\beta ,$$

where the offset *β* and scale factor
*γ* are learnable parameters that are updated during network
training.

[1] Wu, Yuxin, and Kaiming He. “Group Normalization.” Preprint submitted June 11, 2018. https://arxiv.org/abs/1803.08494.

`trainNetwork`

| `trainingOptions`

| `reluLayer`

| `convolution2dLayer`

| `fullyConnectedLayer`

| `batchNormalizationLayer`

| `layerNormalizationLayer`

| `instanceNormalizationLayer`