Decoupled Weight Decay Regularization

Inside the paper

The problem

Adding a squared-weight penalty and shrinking weights directly are often treated as interchangeable ways to regularize training. The paper asks whether that equivalence still holds when an optimizer adapts its update scale separately for each parameter.

The idea

The authors separate the weight-shrinking step from the gradient update. For Adam, this gives a version called AdamW in which the shrinkage is not transformed by the optimizer’s adaptive gradient scaling.

How it works

• Calculate the gradient of the training loss without adding a squared-weight penalty to that gradient.
• Use the optimizer’s usual gradient-based update, including Adam’s running gradient statistics when using AdamW.
• Apply a separate decay proportional to the current weights, controlled by its own regularization setting.
• Compare learning-rate and decay settings across training runs to examine how independently they can be tuned.

What it showed

On the image classification experiments, AdamW produced better generalization than the tested Adam implementation with a squared-weight penalty. The authors also report that the useful decay settings were more separable from learning-rate settings in their comparisons.

Read it for

Read Section 2 for the distinction between the two updates, then Figures 2 and 3 with Sections 4.2 and 4.3 for the experiments. The warm-restart extension in Section 4.4 can wait for a second pass.

Written in our own words from the paper itself. A draft the editors have not yet reviewed.