cyclicalLearnRate

Cyclical learning rate schedule

Since R2024b

Description

A cyclical learning rate schedule periodically increases and decreases the learning rate.

Tip

To easily use a cyclical learning rate schedule with the default options, specify the LearnRateSchedule argument of the trainingOptions function as "cyclical".

Creation

Description

schedule = cyclicalLearnRate creates a cyclicalLearnRate object.

This syntax is equivalent to setting the LearnRateSchedule argument of the trainingOptions function to "cyclical".

schedule = cyclicalLearnRate(Name=Value) specifies additional properties using one or more name-value arguments. For example, MaxFactor=2 specifies a maximum scaling factor of 2.

Properties

MaxFactor — Maximum scaling factor

5 (default) | positive scalar

Maximum scaling factor, specified as a positive scalar.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

Period — Number of steps in cycle

10 (default) | positive integer

Number of steps in the cycle, specified as a positive integer.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

StepRatio — Ratio of time spent increasing versus decreasing learning rate

0.5 (default) | scalar in the range (0,1)

Ratio of time spent increasing versus decreasing the learning rate, specified as a scalar in the range (0,1).

Data Types: single | double

FrequencyUnit — Frequency unit

"epoch" (default) | "iteration"

Frequency unit, specified as "epoch" or "iteration".

If FrequencyUnit is "epoch", then the software updates the learning rate every epoch and each iteration of the epoch uses the same learning rate. If FrequencyUnit is "iteration", then the software updates the learning rate every iteration.

NumSteps — Number of steps to drop learning rate

Inf (default)

This property is read-only.

Number of steps to drop learning rate, specified as Inf.

cyclicalLearnRate objects are infinite learning rate schedules, so NumSteps is Inf.

Data Types: double

Examples

Create Cyclical Learning Rate Schedule

Create a cyclical learning rate schedule with the default settings.

schedule = cyclicalLearnRate

schedule = 
  cyclicalLearnRate with properties:

        MaxFactor: 5
           Period: 10
        StepRatio: 0.5000
    FrequencyUnit: "epoch"
         NumSteps: Inf

Specify this schedule as a training option.

options = trainingOptions("adam",LearnRateSchedule=schedule);

As an alternative to creating a cyclical learning rate schedule object, to easily use a cyclical learning rate schedule with the default options, specify the LearnRateSchedule argument of the trainingOptions function as "piecewise".

options = trainingOptions("adam",LearnRateSchedule="cyclical");

Customize Cyclical Learning Rate Schedule

Create a cyclical learning rate schedule that cycles for a period of 100 iterations.

schedule = cyclicalLearnRate( ...
    Period=100, ...
    FrequencyUnit="iteration")

schedule = 
  cyclicalLearnRate with properties:

        MaxFactor: 5
           Period: 100
        StepRatio: 0.5000
    FrequencyUnit: "iteration"
         NumSteps: Inf

Specify this schedule as a training option.

options = trainingOptions("adam",LearnRateSchedule=schedule);

Algorithms

Cyclical Learning Rate Schedule

A cyclical learning rate schedule periodically increases and decreases the learning rate.

The software uses this formula to calculate the learning rate:

$$\alpha =\{\begin{array}{cc}{\alpha }_0\left(1+({\gamma }_N-1)\frac{(k-1)\mathrm{mod}P}{\lceil qP\rceil }\right)& \text{if }(k-1\mathrm{mod}P)<\lceil qP\rceil \\ {\alpha }_0\left({\gamma }_N-\left({\gamma }_N-1\right)\frac{\left((k-1)\mathrm{mod}P\right)-\lceil qP\rceil }{P-\lceil qP\rceil }\right)& \text{if }(k-1\mathrm{mod}P)\ge \lceil qP\rceil \end{array},$$

where:

• α₀ is the base learning rate, specified by the InitialLearnRate option of the trainingOptions function.

• γ_N is the maximum scaling factor, specified by the MaxFactor argument.

• P is the period, specified by the Period argument.

• q is the step ratio, specified by the StepRatio argument.

If FrequencyUnit is "iteration", then the variable k denotes the schedule iteration number. The value of k can be different from the training iteration number. For example, if you train using {"warmup",schedule}, where schedule is a cyclicalLearnRate object with FrequencyUnit set to "iteration", then at training iteration 15, the cyclicalLearnRate object uses k with a value of 10 because the warm-up learning rate schedule runs for 5 training iterations.

If FrequencyUnit is "epoch", then the variable k denotes the schedule epoch number. In this case, the schedule uses the same learning rate for each iteration of the epoch. The value of k can be different from the training epoch number. For example, if you train using {warmupLearnRate(FrequencyUnit="epoch"),schedule}, where schedule is a cyclicalLearnRate object with FrequencyUnit set to "epoch", then at training epoch 15, the cyclicalLearnRate object uses k with a value of 10 because the warm-up learning rate schedule processes the first 5 epochs.

This plot shows an overview of the cyclical learning rate schedule with the default settings.

Version History

Introduced in R2024b