PyTorch Regularization, Learning Rate Schedulers and Metrics

Regularization Toolkit

Training loss alone does not tell you whether a model is useful. A model can memorize the training set and fail on new data. Regularization and validation metrics help you build models that generalize.

This lesson covers practical controls developers use every day: dropout, weight decay, augmentation, early stopping, learning rate schedulers, metric tracking, and checkpoint selection based on validation performance.

Regularization reduces overfitting. The right method depends on the problem. Image models often benefit from augmentation. Dense networks may need dropout and weight decay. Large pretrained models often need careful learning rates more than heavy dropout.

Use dropout inside the model for neural feature regularization.
Use weight decay in AdamW or SGD to discourage large weights.
Use data augmentation when the input domain supports label-preserving changes.
Use early stopping when validation loss stops improving.

Learning Rate Schedules

A fixed learning rate is a baseline, not always the best final choice. Schedulers reduce or reshape the learning rate during training. This can stabilize convergence and improve final accuracy.

StepLR reduces learning rate after fixed intervals.
ReduceLROnPlateau reacts to validation metrics.
CosineAnnealingLR gradually lowers learning rate in a smooth curve.

Scheduler and Early Stopping Pattern

This pattern saves the best model and stops when validation loss stops improving.

Scheduler and Early Stopping Pattern

best_val_loss = float("inf")
patience = 5
bad_epochs = 0

optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=1e-4)
scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(
    optimizer,
    mode="min",
    factor=0.5,
    patience=2,
)

for epoch in range(1, 51):
    train_loss = train_one_epoch(model, train_loader, optimizer, loss_fn, device)
    val_loss, val_acc = evaluate(model, val_loader, loss_fn, device)
    scheduler.step(val_loss)

    if val_loss < best_val_loss:
        best_val_loss = val_loss
        bad_epochs = 0
        torch.save(model.state_dict(), "best_model.pt")
    else:
        bad_epochs += 1

    if bad_epochs >= patience:
        print("Early stopping")
        break

ReduceLROnPlateau uses a validation metric, so call scheduler.step(val_loss).
Save the best validation model, not the last epoch by habit.

Step a Learning Rate Schedule at the Intended Frequency

import torch

parameter = torch.nn.Parameter(torch.tensor(1.0))
optimizer = torch.optim.SGD([parameter], lr=0.1, weight_decay=0.01)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=2, gamma=0.5)

for epoch in range(4):
    optimizer.step()
    scheduler.step()
    print(epoch + 1, optimizer.param_groups[0]['lr'])

Output

Before you move on