Policy Gradient Loss
Policy Gradient Loss
The REINFORCE algorithm uses the policy gradient theorem to directly optimize a stochastic policy. The loss is constructed so that gradient descent increases the probability of actions that led to higher-than-average returns and decreases the probability of actions that led to lower-than-average returns.
Given the log-probabilities of the actions taken, the rewards at each timestep, and a discount factor gamma, compute the policy gradient loss with a mean-return baseline.
Algorithm
- Compute discounted returns backward:
- Subtract the mean return as a baseline to reduce variance:
- Compute the loss (negative because we want gradient ascent on expected return):
Examples
Input:
log_probs = [-1.0, -2.0, -0.5], rewards = [1, 1, 1], gamma = 1.0
Output:
0.1667
Returns: G = [3, 2, 1]. Mean = 2. Advantages = [1, 0, -1]. Loss = -(-11 + -20 + -0.5*(-1))/3 = -(- 1 + 0 + 0.5)/3 = 0.5/3.
Input:
log_probs = [-1.0, -0.5], rewards = [0, 10], gamma = 0.0
Output:
-1.25
With gamma = 0, returns equal immediate rewards: G = [0, 10]. Mean = 5. Advantages = [-5, 5]. Loss = -(-1*(-5) + -0.5*5)/2 = -(5 - 2.5)/2 = -1.25.
Hint 1
First compute returns G backward (G[T-1] = rewards[T-1], G[t] = rewards[t] + gamma * G[t+1]). Then compute mean_G = sum(G) / T. Then advantages = [g - mean_G for g in G]. Finally loss = -sum(lp * a for lp, a in zip(log_probs, advantages)) / T.
Hint 2
The negative sign is critical. Without it, gradient descent would decrease the expected return instead of increasing it.
Requirements
- Compute discounted returns using backward recursion
- Subtract the mean return from each return to get advantages
- Compute loss as the negative mean of (log_prob * advantage)
- Return a single float
Constraints
- log_probs and rewards have the same length (at least 1)
- log_probs contains negative floats (log of probabilities)
- 0 <= gamma <= 1
- Return a single float
- Time limit: 300 ms
Log in to take notes on this problem
Accepts: array
Accepts: array
Accepts: number
Policy Gradient Loss
Policy Gradient Loss
The REINFORCE algorithm uses the policy gradient theorem to directly optimize a stochastic policy. The loss is constructed so that gradient descent increases the probability of actions that led to higher-than-average returns and decreases the probability of actions that led to lower-than-average returns.
Given the log-probabilities of the actions taken, the rewards at each timestep, and a discount factor gamma, compute the policy gradient loss with a mean-return baseline.
Algorithm
- Compute discounted returns backward:
- Subtract the mean return as a baseline to reduce variance:
- Compute the loss (negative because we want gradient ascent on expected return):
Examples
Input:
log_probs = [-1.0, -2.0, -0.5], rewards = [1, 1, 1], gamma = 1.0
Output:
0.1667
Returns: G = [3, 2, 1]. Mean = 2. Advantages = [1, 0, -1]. Loss = -(-11 + -20 + -0.5*(-1))/3 = -(- 1 + 0 + 0.5)/3 = 0.5/3.
Input:
log_probs = [-1.0, -0.5], rewards = [0, 10], gamma = 0.0
Output:
-1.25
With gamma = 0, returns equal immediate rewards: G = [0, 10]. Mean = 5. Advantages = [-5, 5]. Loss = -(-1*(-5) + -0.5*5)/2 = -(5 - 2.5)/2 = -1.25.
Hint 1
First compute returns G backward (G[T-1] = rewards[T-1], G[t] = rewards[t] + gamma * G[t+1]). Then compute mean_G = sum(G) / T. Then advantages = [g - mean_G for g in G]. Finally loss = -sum(lp * a for lp, a in zip(log_probs, advantages)) / T.
Hint 2
The negative sign is critical. Without it, gradient descent would decrease the expected return instead of increasing it.
Requirements
- Compute discounted returns using backward recursion
- Subtract the mean return from each return to get advantages
- Compute loss as the negative mean of (log_prob * advantage)
- Return a single float
Constraints
- log_probs and rewards have the same length (at least 1)
- log_probs contains negative floats (log of probabilities)
- 0 <= gamma <= 1
- Return a single float
- Time limit: 300 ms
Log in to take notes on this problem
Accepts: array
Accepts: array
Accepts: number