Replay Buffer Sample
Replay Buffer Sample
A replay buffer (or experience replay memory) stores past transitions that an agent has experienced. During training, a random batch of transitions is sampled from the buffer to break temporal correlations and stabilize learning. This technique is used in DQN, SAC, DDPG, and many other off-policy algorithms.
Given a buffer of transitions, a batch size, and a random seed, sample a batch of transitions uniformly at random without replacement.
Algorithm
-
Set the random seed for reproducibility
-
Sample batch_size transitions from the buffer uniformly at random without replacement
Examples
Input:
buffer = [[0, 0, 1.0, 1, 0], [1, 1, 0.5, 2, 0], [2, 0, -1.0, 3, 1], [3, 1, 2.0, 4, 0], [4, 0, 0.0, 0, 1]], batch_size = 3, seed = 42
Output:
[[4, 0, 0.0, 0, 1], [2, 0, -1.0, 3, 1], [3, 1, 2.0, 4, 0]]
With seed 42, random.sample selects indices 4, 2, and 3 from the buffer. The result is a list of 3 transitions with no duplicates.
Input:
buffer = [[0, 0, 1.0, 1, 0], [1, 1, 0.5, 2, 0], [2, 0, -1.0, 3, 1], [3, 1, 2.0, 4, 0], [4, 0, 0.0, 0, 1]], batch_size = 1, seed = 7
Output:
[[4, 0, 0.0, 0, 1]]
A single transition is sampled from the buffer.
Hint 1
np.random.RandomState lets you create a seeded random number generator. Look into its choice method for sampling.
Hint 2
Think about whether the order of your sampled transitions matters for consistent output.
Requirements
- Use NumPy for random sampling with the given seed
- Sample without replacement (no duplicate transitions in the batch)
- Return a list of transitions in a deterministic order
Constraints
- batch_size <= len(buffer)
- buffer is a list of lists (each inner list is a transition)
- seed is an integer
- Return a list of transitions (list of lists)
- Time limit: 300 ms
Try Similar Problems
Log in to take notes on this problem
Accepts: array
Accepts: number
Accepts: number
Replay Buffer Sample
Replay Buffer Sample
A replay buffer (or experience replay memory) stores past transitions that an agent has experienced. During training, a random batch of transitions is sampled from the buffer to break temporal correlations and stabilize learning. This technique is used in DQN, SAC, DDPG, and many other off-policy algorithms.
Given a buffer of transitions, a batch size, and a random seed, sample a batch of transitions uniformly at random without replacement.
Algorithm
-
Set the random seed for reproducibility
-
Sample batch_size transitions from the buffer uniformly at random without replacement
Examples
Input:
buffer = [[0, 0, 1.0, 1, 0], [1, 1, 0.5, 2, 0], [2, 0, -1.0, 3, 1], [3, 1, 2.0, 4, 0], [4, 0, 0.0, 0, 1]], batch_size = 3, seed = 42
Output:
[[4, 0, 0.0, 0, 1], [2, 0, -1.0, 3, 1], [3, 1, 2.0, 4, 0]]
With seed 42, random.sample selects indices 4, 2, and 3 from the buffer. The result is a list of 3 transitions with no duplicates.
Input:
buffer = [[0, 0, 1.0, 1, 0], [1, 1, 0.5, 2, 0], [2, 0, -1.0, 3, 1], [3, 1, 2.0, 4, 0], [4, 0, 0.0, 0, 1]], batch_size = 1, seed = 7
Output:
[[4, 0, 0.0, 0, 1]]
A single transition is sampled from the buffer.
Hint 1
np.random.RandomState lets you create a seeded random number generator. Look into its choice method for sampling.
Hint 2
Think about whether the order of your sampled transitions matters for consistent output.
Requirements
- Use NumPy for random sampling with the given seed
- Sample without replacement (no duplicate transitions in the batch)
- Return a list of transitions in a deterministic order
Constraints
- batch_size <= len(buffer)
- buffer is a list of lists (each inner list is a transition)
- seed is an integer
- Return a list of transitions (list of lists)
- Time limit: 300 ms
Try Similar Problems
Log in to take notes on this problem
Accepts: array
Accepts: number
Accepts: number