Problems

1 / 1

Prioritized Experience Replay

Reinforcement Learning

Easy

Prioritized Experience Replay (PER) improves on uniform replay by sampling transitions proportional to their TD error magnitude. Transitions where the agent was most "surprised" (large TD error) are replayed more often, leading to faster learning. To correct for the bias introduced by non-uniform sampling, importance sampling weights are applied.

Given a list of priority values, an alpha parameter (controls how much prioritization is used), and a beta parameter (controls importance sampling correction), compute the sampling probabilities and normalized importance sampling weights.

Algorithm

Compute powered priorities:

p_i^{\alpha}

Compute sampling probabilities:

P(i) = \frac{p_i^{\alpha}}{\sum_j p_j^{\alpha}}

Compute raw importance sampling weights:

w_i = (N \cdot P(i))^{-\beta}

Normalize weights by the maximum weight:

\hat{w}_i = \frac{w_i}{\max_j w_j}

Loading visualization...

Examples

Input:

priorities = [1, 2, 3], alpha = 1.0, beta = 1.0

Output:

[[0.1667, 0.3333, 0.5], [1.0, 0.5, 0.3333]]

With alpha = 1, probabilities are proportional to priorities. Weight for the least-likely sample (index 0) is largest and normalizes to 1.0. Higher probability samples get smaller weights to correct for oversampling.

Input:

priorities = [1, 1, 1], alpha = 1.0, beta = 1.0

Output:

[[0.3333, 0.3333, 0.3333], [1.0, 1.0, 1.0]]

Equal priorities produce uniform probabilities and all weights equal 1.0.

Hint 1

First compute powered = [p ** alpha for p in priorities]. Then total = sum(powered), probs = [x / total for x in powered]. Then raw_weights = [(n * pr) ** (-beta) for pr in probs] where n = len(priorities). Finally normalize by max(raw_weights).

Hint 2

When alpha = 0, all powered values equal 1, so probabilities become uniform regardless of the original priorities.

Requirements

Raise each priority to the power alpha to control prioritization strength
Compute sampling probabilities by normalizing the powered priorities
Compute importance sampling weights as (N * P(i))^(-beta)
Normalize weights by dividing by the maximum weight
Return a list of two lists: [probabilities, normalized_weights]

Constraints

priorities has at least one element with all positive values
alpha >= 0, beta >= 0
Return [probabilities, normalized_weights] where each is a list of floats
Time limit: 300 ms

Try Similar Problems

Replay Buffer Sample Q Learning Update Sarsa Update Epsilon Greedy Td Value Update

Prev Next

Prioritized Experience Replay

Reinforcement Learning

Easy

Algorithm

Compute powered priorities:

p_i^{\alpha}

Compute sampling probabilities:

P(i) = \frac{p_i^{\alpha}}{\sum_j p_j^{\alpha}}

Compute raw importance sampling weights:

w_i = (N \cdot P(i))^{-\beta}

Normalize weights by the maximum weight:

\hat{w}_i = \frac{w_i}{\max_j w_j}

Loading visualization...

Examples

Input:

priorities = [1, 2, 3], alpha = 1.0, beta = 1.0

Output:

[[0.1667, 0.3333, 0.5], [1.0, 0.5, 0.3333]]

Input:

priorities = [1, 1, 1], alpha = 1.0, beta = 1.0

Output:

[[0.3333, 0.3333, 0.3333], [1.0, 1.0, 1.0]]

Equal priorities produce uniform probabilities and all weights equal 1.0.

Hint 1

Hint 2

When alpha = 0, all powered values equal 1, so probabilities become uniform regardless of the original priorities.

Requirements

Raise each priority to the power alpha to control prioritization strength
Compute sampling probabilities by normalizing the powered priorities
Compute importance sampling weights as (N * P(i))^(-beta)
Normalize weights by dividing by the maximum weight
Return a list of two lists: [probabilities, normalized_weights]

Constraints

priorities has at least one element with all positive values
alpha >= 0, beta >= 0
Return [probabilities, normalized_weights] where each is a list of floats
Time limit: 300 ms

Try Similar Problems

Replay Buffer Sample Q Learning Update Sarsa Update Epsilon Greedy Td Value Update

Case 1

Case 2

priorities =

Accepts: array

alpha =

Accepts: number

beta =

Accepts: number