Gradient Clipping (Global Norm)
Gradient Clipping (Global Norm)
Implement global norm gradient clipping to prevent exploding gradients in deep networks. Scale gradients proportionally when their norm exceeds a threshold.
Compute Gradient Norm:
∣∣g∣∣=i∑gi2Clipping Rule:
gclipped={gg⋅∣∣g∣∣max_normif ∣∣g∣∣≤max_normotherwiseWhere: g = gradient vector, ||g|| = L2 norm, max_norm = clipping threshold
Function Arguments
g: np.ndarray- Gradient array (any shape)max_norm: float- Maximum allowed norm (positive)
Examples
Input: g=[0.1, 0.2, 0.2], max_norm=1.0
Output: [0.1, 0.2, 0.2]
norm = 0.3 ≤ 1.0, so no clipping needed
Input: g=[6, 8], max_norm=5.0
Output: [3.0, 4.0]
norm = 10.0 > 5.0, so multiply by 5.0/10.0 = 0.5
Input: g=[[2, 2], [2, 2]], max_norm=2.0
Output: [[1.0, 1.0], [1.0, 1.0]]
norm = 4.0 > 2.0, so multiply by 2.0/4.0 = 0.5
Hint 1
Use np.asarray() to convert input and np.linalg.norm() to compute the L2 norm of the entire gradient array.
Hint 2
Check if norm <= max_norm to decide whether clipping is needed. Use g.copy() to avoid in-place modification.
Hint 3
Scale gradients with g * (max_norm / norm). Handle edge cases: zero norm or non-positive max_norm should return unchanged gradients.
Requirements
- Return
np.ndarraywith same shape as input - Handle any array shape (1D, 2D, 3D, etc.)
- Handle edge cases: zero norm, negative max_norm
- Preserve gradient direction, only scale magnitude
Constraints
- Gradient arrays up to 10⁶ elements
- Numerical precision: relative tolerance ≤ 1e-8
- Libraries: NumPy only
Log in to take notes on this problem
Accepts: array
Accepts: number
Gradient Clipping (Global Norm)
Gradient Clipping (Global Norm)
Implement global norm gradient clipping to prevent exploding gradients in deep networks. Scale gradients proportionally when their norm exceeds a threshold.
Compute Gradient Norm:
∣∣g∣∣=i∑gi2Clipping Rule:
gclipped={gg⋅∣∣g∣∣max_normif ∣∣g∣∣≤max_normotherwiseWhere: g = gradient vector, ||g|| = L2 norm, max_norm = clipping threshold
Function Arguments
g: np.ndarray- Gradient array (any shape)max_norm: float- Maximum allowed norm (positive)
Examples
Input: g=[0.1, 0.2, 0.2], max_norm=1.0
Output: [0.1, 0.2, 0.2]
norm = 0.3 ≤ 1.0, so no clipping needed
Input: g=[6, 8], max_norm=5.0
Output: [3.0, 4.0]
norm = 10.0 > 5.0, so multiply by 5.0/10.0 = 0.5
Input: g=[[2, 2], [2, 2]], max_norm=2.0
Output: [[1.0, 1.0], [1.0, 1.0]]
norm = 4.0 > 2.0, so multiply by 2.0/4.0 = 0.5
Hint 1
Use np.asarray() to convert input and np.linalg.norm() to compute the L2 norm of the entire gradient array.
Hint 2
Check if norm <= max_norm to decide whether clipping is needed. Use g.copy() to avoid in-place modification.
Hint 3
Scale gradients with g * (max_norm / norm). Handle edge cases: zero norm or non-positive max_norm should return unchanged gradients.
Requirements
- Return
np.ndarraywith same shape as input - Handle any array shape (1D, 2D, 3D, etc.)
- Handle edge cases: zero norm, negative max_norm
- Preserve gradient direction, only scale magnitude
Constraints
- Gradient arrays up to 10⁶ elements
- Numerical precision: relative tolerance ≤ 1e-8
- Libraries: NumPy only
Log in to take notes on this problem
Accepts: array
Accepts: number