Problems
Loading...
1 / 1

Implement AdamW (Decoupled Weight Decay)

Optimization
Easy

Implement one update step of AdamW optimizer. Given current parameters, moments, and gradients, return updated parameters and moments using decoupled weight decay.

Step 1: Update First Moment

mt=β1mt−1+(1−β1)gtm_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t

Step 2: Update Second Moment

vt=β2vt−1+(1−β2)gt2v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2

Step 3: AdamW Parameter Update

wt=wt−1−η(decay⋅wt−1)−η⋅mtvt+εw_t = w_{t-1} - \eta(\text{decay} \cdot w_{t-1}) - \eta \cdot \frac{m_t}{\sqrt{v_t} + \varepsilon}

Where: w = parameters, m = first moment, v = second moment, g = gradients, \u03b7 = learning rate, decay = weight decay

Loading visualization...

Examples

Input: w=[1.0, -2.0], m=[0, 0], v=[0, 0], grad=[0.3, -0.7], lr=0.01, weight_decay=0.1

Output: ([0.967, -1.966], [0.03, -0.07], [0.00009, 0.00049])

Input: w=[1.0, 2.0], m=[0.1, 0.2], v=[0.01, 0.04], grad=[0, 0], lr=0.01, weight_decay=0.1

Output: ([0.99, 1.989], [0.09, 0.18], [0.00999, 0.03996])

Hint 1

Convert inputs to NumPy arrays first. Update moments using exponential moving averages.

Hint 2

The parameter update has two parts: weight decay term and adaptive gradient term. Apply both simultaneously.

Requirements

  • Return tuple (new_w, new_m, new_v) with same shapes as inputs
  • Vectorized implementation only (no Python loops)
  • Handle any array shape (1D, 2D, etc.)

Constraints

  • 0 < beta1, beta2 < 1
  • lr > 0, weight_decay ≥ 0
  • Libraries: NumPy only
Try Similar Problems
Adam OptimizerNesterov MomentumNadam OptimizerAdagrad OptimizerRmsprop Optimizer
PrevNext