Learning Rate Scheduling
Published Jan 27, 2025
Contribute to Docs
Learning Rate Scheduling is a method to adjust the learning rate during the training process of a neural network. The learning rate defines the step size used by optimization algorithms, such as Stochastic Gradient Descent (SGD), to minimize the loss function. By modifying the learning rate throughout training, a schedule can help models converge faster and achieve better accuracy.
Key Points
- Higher learning rate initially: Helps the model explore parameter values quickly during early training.
- Gradual reduction: Allows fine-tuning with smaller steps as the model approaches optimal parameters.
- Types of schedules:
- Step-based schedules: Reduce the learning rate at fixed intervals (e.g., every 10 epochs). This approach is simple and works well for many cases.
- Exponential schedules: Reduce the learning rate by a fixed factor (e.g., multiplying it by 0.9) after every epoch or step. This ensures a smooth, gradual decay over time.
- Adaptive schedules: Automatically adjust the learning rate based on the model’s performance (e.g., reducing the learning rate when the validation loss stops improving). These are more dynamic and can adapt to the needs of the model during training.
Syntax
Here’s a general syntax for implementing a learning rate schedule in PyTorch:
torch.optim.lr_scheduler.<SchedulerClass>(
optimizer,
step_size=<int>,
gamma=<float>,
<other_parameters>
)
optimizer
: The optimizer whose learning rate will be updated.step_size
: Interval (in epochs) at which the learning rate will be reduced. An epoch refers to one complete pass through the entire training dataset by the model. During each epoch, the model processes all training examples and updates its weights accordingly.gamma
: Multiplicative factor by which the learning rate is reduced.- : Additional parameters specific to the chosen scheduler.
Example
The following example demonstrates the use of a StepLR Learning Rate Scheduler in PyTorch to reduce the learning rate by a factor of 0.9 every 10 epochs:
import torchimport torch.nn as nnimport torch.optim as optim# Define a simple neural networkclass SimpleNN(nn.Module):def __init__(self):super(SimpleNN, self).__init__()self.fc = nn.Linear(10, 1)def forward(self, x):return self.fc(x)# Initialize the model, loss function, and optimizermodel = SimpleNN()criterion = nn.MSELoss()optimizer = optim.SGD(model.parameters(), lr=0.1)# Define the learning rate schedulerscheduler = optim.lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.9)# Training loopfor epoch in range(30): # Train for 30 epochs# Simulate training stepinputs = torch.randn(5, 10)targets = torch.randn(5, 1)optimizer.zero_grad()outputs = model(inputs)loss = criterion(outputs, targets)loss.backward()optimizer.step()# Update the learning ratescheduler.step()# Print learning rateprint(f"Epoch {epoch + 1}: Learning Rate = {scheduler.get_last_lr()[0]:.5f}")
The code above produces the output as below:
Epoch 1: Learning Rate = 0.10000Epoch 2: Learning Rate = 0.10000Epoch 3: Learning Rate = 0.10000Epoch 4: Learning Rate = 0.10000Epoch 5: Learning Rate = 0.10000Epoch 6: Learning Rate = 0.10000Epoch 7: Learning Rate = 0.10000Epoch 8: Learning Rate = 0.10000Epoch 9: Learning Rate = 0.10000Epoch 10: Learning Rate = 0.09000Epoch 11: Learning Rate = 0.09000Epoch 12: Learning Rate = 0.09000Epoch 13: Learning Rate = 0.09000Epoch 14: Learning Rate = 0.09000Epoch 15: Learning Rate = 0.09000Epoch 16: Learning Rate = 0.09000Epoch 17: Learning Rate = 0.09000Epoch 18: Learning Rate = 0.09000Epoch 19: Learning Rate = 0.09000Epoch 20: Learning Rate = 0.08100Epoch 21: Learning Rate = 0.08100Epoch 22: Learning Rate = 0.08100Epoch 23: Learning Rate = 0.08100Epoch 24: Learning Rate = 0.08100Epoch 25: Learning Rate = 0.08100Epoch 26: Learning Rate = 0.08100Epoch 27: Learning Rate = 0.08100Epoch 28: Learning Rate = 0.08100Epoch 29: Learning Rate = 0.08100Epoch 30: Learning Rate = 0.07290
Contribute to Docs
- Learn more about how to get involved.
- Edit this page on GitHub to fix an error or make an improvement.
- Submit feedback to let us know how we can improve Docs.
Learn AI on Codecademy
- Skill path
Build a Machine Learning Model
Learn to build machine learning models with Python.Includes 10 CoursesWith CertificateBeginner Friendly23 hours - Career path
Data Scientist: Machine Learning Specialist
Machine Learning Data Scientists solve problems at scale, make predictions, find patterns, and more! They use Python, SQL, and algorithms.Includes 27 CoursesWith Professional CertificationBeginner Friendly90 hours