Problems

1 / 1

Implement AdaDelta Update Step

Optimization

Medium

Implement one update step of AdaDelta optimizer. Given current parameters, gradients, and running averages, return updated parameters and averages using the adaptive learning rate approach.

Step 1: Update Squared Gradient Average

E[g^2]_t = \rho E[g^2]_{t-1} + (1 - \rho) g_t^2

Step 2: Compute Parameter Update

\Delta w_t = -\frac{\sqrt{E[\Delta w^2]_{t-1} + \varepsilon}}{\sqrt{E[g^2]_t + \varepsilon}} \cdot g_t

Step 3: Update Squared Update Average

E[\Delta w^2]_t = \rho E[\Delta w^2]_{t-1} + (1 - \rho) (\Delta w_t)^2

Step 4: Update Parameters

w_t = w_{t-1} + \Delta w_t

Where: w = parameters, g = gradients, E[g²] = running avg of squared gradients, E[Δw²] = running avg of squared updates, ρ = decay rate, ε = stability constant

Function Arguments

w: np.ndarray - Current parameters (any shape)
grad: np.ndarray - Current gradients (same shape as w)
E_grad_sq: np.ndarray - Running average of squared gradients (same shape as w)
E_update_sq: np.ndarray - Running average of squared updates (same shape as w)
rho: float = 0.9 - Decay rate for moving averages
eps: float = 1e-6 - Small constant for numerical stability

Loading visualization...

Examples

Input: w=[1.0, -1.0], grad=[0.1, -0.2], E_grad_sq=[0.01, 0.04], E_update_sq=[0.001, 0.004]

Output: ([0.968, -0.937], [0.01, 0.04], [0.001, 0.004])

Updates computed using ratio of RMS update to RMS gradient

Input: w=[1.0, 2.0], grad=[0.0, 0.0], E_grad_sq=[0.01, 0.04], E_update_sq=[0.001, 0.004]

Output: ([1.0, 2.0], [0.009, 0.036], [0.0009, 0.0036])

Zero gradient: no parameter update, but averages still decay

Input: w=[1.0, 2.0], grad=[0.1, 0.2], E_grad_sq=[0.0, 0.0], E_update_sq=[0.0, 0.0]

Output: ([0.997, 1.997], [0.001, 0.004], [0.000001, 0.000001])

First step: small updates due to near-zero accumulated update history

Hint 1

Convert inputs to NumPy arrays first. Update E_grad_sq using exponential moving average of squared gradients.

Hint 2

Compute delta_w using the ratio of RMS update to RMS gradient, then update E_update_sq with squared delta_w.

Requirements

Return tuple (new_w, new_E_grad_sq, new_E_update_sq) with same shapes as inputs
Use the exact update formulas above
No manual learning rate parameter (AdaDelta eliminates it)
Vectorized implementation only (no Python loops)
Handle any array shape (1D, 2D, etc.)

Constraints

0 < rho < 1
eps > 0 (small positive float)
Libraries: NumPy only

Try Similar Problems

Adagrad Optimizer Rmsprop Optimizer Adam Optimizer Gradient Descent Quadratic Adamw Optimizer

Prev Next

Implement AdaDelta Update Step

Optimization

Medium

Implement one update step of AdaDelta optimizer. Given current parameters, gradients, and running averages, return updated parameters and averages using the adaptive learning rate approach.

Step 1: Update Squared Gradient Average

E[g^2]_t = \rho E[g^2]_{t-1} + (1 - \rho) g_t^2

Step 2: Compute Parameter Update

\Delta w_t = -\frac{\sqrt{E[\Delta w^2]_{t-1} + \varepsilon}}{\sqrt{E[g^2]_t + \varepsilon}} \cdot g_t

Step 3: Update Squared Update Average

E[\Delta w^2]_t = \rho E[\Delta w^2]_{t-1} + (1 - \rho) (\Delta w_t)^2

Step 4: Update Parameters

w_t = w_{t-1} + \Delta w_t

Where: w = parameters, g = gradients, E[g²] = running avg of squared gradients, E[Δw²] = running avg of squared updates, ρ = decay rate, ε = stability constant

Function Arguments

w: np.ndarray - Current parameters (any shape)
grad: np.ndarray - Current gradients (same shape as w)
E_grad_sq: np.ndarray - Running average of squared gradients (same shape as w)
E_update_sq: np.ndarray - Running average of squared updates (same shape as w)
rho: float = 0.9 - Decay rate for moving averages
eps: float = 1e-6 - Small constant for numerical stability

Loading visualization...

Examples

Input: w=[1.0, -1.0], grad=[0.1, -0.2], E_grad_sq=[0.01, 0.04], E_update_sq=[0.001, 0.004]

Output: ([0.968, -0.937], [0.01, 0.04], [0.001, 0.004])

Updates computed using ratio of RMS update to RMS gradient

Input: w=[1.0, 2.0], grad=[0.0, 0.0], E_grad_sq=[0.01, 0.04], E_update_sq=[0.001, 0.004]

Output: ([1.0, 2.0], [0.009, 0.036], [0.0009, 0.0036])

Zero gradient: no parameter update, but averages still decay

Input: w=[1.0, 2.0], grad=[0.1, 0.2], E_grad_sq=[0.0, 0.0], E_update_sq=[0.0, 0.0]

Output: ([0.997, 1.997], [0.001, 0.004], [0.000001, 0.000001])

First step: small updates due to near-zero accumulated update history

Hint 1

Convert inputs to NumPy arrays first. Update E_grad_sq using exponential moving average of squared gradients.

Hint 2

Compute delta_w using the ratio of RMS update to RMS gradient, then update E_update_sq with squared delta_w.

Requirements

Return tuple (new_w, new_E_grad_sq, new_E_update_sq) with same shapes as inputs
Use the exact update formulas above
No manual learning rate parameter (AdaDelta eliminates it)
Vectorized implementation only (no Python loops)
Handle any array shape (1D, 2D, etc.)

Constraints

0 < rho < 1
eps > 0 (small positive float)
Libraries: NumPy only

Try Similar Problems

Adagrad Optimizer Rmsprop Optimizer Adam Optimizer Gradient Descent Quadratic Adamw Optimizer

Case 1

Case 2

Case 3

w =

Accepts: array

grad =

Accepts: array

E_grad_sq =

Accepts: array

E_update_sq =

Accepts: array

rho =

Accepts: number

eps =

Accepts: number