He Initialization
He Initialization
He (Kaiming) initialization is designed for networks using ReLU activations. Since ReLU zeros out negative values (roughly half the distribution), He initialization uses a larger variance than Xavier to compensate, depending only on the fan-in.
Given a raw weight matrix W with values in [0, 1] and fan_in (number of input units), scale the weights to He uniform initialization.
Algorithm
- Compute the He uniform bound:
- Map each raw weight from [0, 1] to [-L, L]:
Examples
Input:
W = [[0.5, 0.5]], fan_in = 2
Output:
[[0.0, 0.0]]
With fan_in = 2, limit = sqrt(6/2) = sqrt(3) = 1.7321. Raw value 0.5 maps to 0.5 * 2 * 1.7321 - 1.7321 = 0.
Input:
W = [[0], [1]], fan_in = 2
Output:
[[-1.7321], [1.7321]]
Raw 0 maps to -limit and raw 1 maps to +limit. The wider range compared to Xavier compensates for ReLU killing half the activations.
Hint 1
Compute limit = sqrt(6 / fan_in). Then for each element: scaled = raw * 2 * limit - limit. Note that fan_out is not needed since He init only depends on fan_in.
Hint 2
Compare with Xavier: Xavier uses sqrt(6 / (fan_in + fan_out)) while He uses sqrt(6 / fan_in). He's limit is always larger, giving a wider initialization range for ReLU networks.
Requirements
- Scale raw uniform [0, 1] weights to He uniform range [-limit, limit]
- Use the He uniform formula: limit = sqrt(6 / fan_in)
- Return the scaled weight matrix as a list of lists of floats
Constraints
- W has at least 1 row and 1 column with values in [0, 1]
- fan_in >= 1
- Return a list of lists of floats with the same shape as W
- Time limit: 300 ms
Log in to take notes on this problem
Accepts: array
Accepts: number
He Initialization
He Initialization
He (Kaiming) initialization is designed for networks using ReLU activations. Since ReLU zeros out negative values (roughly half the distribution), He initialization uses a larger variance than Xavier to compensate, depending only on the fan-in.
Given a raw weight matrix W with values in [0, 1] and fan_in (number of input units), scale the weights to He uniform initialization.
Algorithm
- Compute the He uniform bound:
- Map each raw weight from [0, 1] to [-L, L]:
Examples
Input:
W = [[0.5, 0.5]], fan_in = 2
Output:
[[0.0, 0.0]]
With fan_in = 2, limit = sqrt(6/2) = sqrt(3) = 1.7321. Raw value 0.5 maps to 0.5 * 2 * 1.7321 - 1.7321 = 0.
Input:
W = [[0], [1]], fan_in = 2
Output:
[[-1.7321], [1.7321]]
Raw 0 maps to -limit and raw 1 maps to +limit. The wider range compared to Xavier compensates for ReLU killing half the activations.
Hint 1
Compute limit = sqrt(6 / fan_in). Then for each element: scaled = raw * 2 * limit - limit. Note that fan_out is not needed since He init only depends on fan_in.
Hint 2
Compare with Xavier: Xavier uses sqrt(6 / (fan_in + fan_out)) while He uses sqrt(6 / fan_in). He's limit is always larger, giving a wider initialization range for ReLU networks.
Requirements
- Scale raw uniform [0, 1] weights to He uniform range [-limit, limit]
- Use the He uniform formula: limit = sqrt(6 / fan_in)
- Return the scaled weight matrix as a list of lists of floats
Constraints
- W has at least 1 row and 1 column with values in [0, 1]
- fan_in >= 1
- Return a list of lists of floats with the same shape as W
- Time limit: 300 ms
Log in to take notes on this problem
Accepts: array
Accepts: number