Text Chunking
Text Chunking
Text chunking splits a sequence of tokens into fixed-size chunks with optional overlap between consecutive chunks. This is a fundamental preprocessing step in NLP pipelines, especially in retrieval-augmented generation (RAG) systems where documents must be split into manageable segments for embedding and retrieval.
Given a list of tokens, a chunk size, and an overlap count, split the tokens into chunks.
Algorithm
- Compute the step size between chunk start positions:
- Starting from position 0, extract a chunk of chunk_size tokens, then advance by step. Stop once a chunk reaches the end of the token list.
Examples
Input:
tokens = ["a", "b", "c", "d", "e", "f"], chunk_size = 3, overlap = 0
Output:
[["a", "b", "c"], ["d", "e", "f"]]
With no overlap and step = 3, the tokens are split into two non-overlapping chunks of size 3.
Input:
tokens = ["a", "b", "c", "d", "e", "f", "g"], chunk_size = 3, overlap = 1
Output:
[["a", "b", "c"], ["c", "d", "e"], ["e", "f", "g"]]
With overlap = 1 and step = 2, each consecutive chunk shares its last token with the next chunk's first token. This overlap provides context continuity.
Hint 1
Compute step = chunk_size - overlap. Then iterate from i = 0, stepping by 'step' each time. At each position, take a slice tokens[i:i+chunk_size]. Stop once the current chunk reaches the end of the list.
Hint 2
Use a for loop: for i in range(0, len(tokens), step). Append tokens[i:i+chunk_size]. Break if i + chunk_size >= len(tokens) to avoid producing redundant trailing chunks that are entirely covered by the previous chunk.
Requirements
- Split the token list into chunks of the specified size with the given overlap between consecutive chunks
- The step between chunk start positions is chunk_size - overlap
- The last chunk may be shorter than chunk_size if there are remaining tokens
- Return a list of lists of tokens
Constraints
- tokens has at least 1 element
- chunk_size >= 1
- 0 <= overlap < chunk_size
- Return a list of lists
- Time limit: 300 ms
Try Similar Problems
Log in to take notes on this problem
Accepts: array
Accepts: number
Accepts: number
Text Chunking
Text Chunking
Text chunking splits a sequence of tokens into fixed-size chunks with optional overlap between consecutive chunks. This is a fundamental preprocessing step in NLP pipelines, especially in retrieval-augmented generation (RAG) systems where documents must be split into manageable segments for embedding and retrieval.
Given a list of tokens, a chunk size, and an overlap count, split the tokens into chunks.
Algorithm
- Compute the step size between chunk start positions:
- Starting from position 0, extract a chunk of chunk_size tokens, then advance by step. Stop once a chunk reaches the end of the token list.
Examples
Input:
tokens = ["a", "b", "c", "d", "e", "f"], chunk_size = 3, overlap = 0
Output:
[["a", "b", "c"], ["d", "e", "f"]]
With no overlap and step = 3, the tokens are split into two non-overlapping chunks of size 3.
Input:
tokens = ["a", "b", "c", "d", "e", "f", "g"], chunk_size = 3, overlap = 1
Output:
[["a", "b", "c"], ["c", "d", "e"], ["e", "f", "g"]]
With overlap = 1 and step = 2, each consecutive chunk shares its last token with the next chunk's first token. This overlap provides context continuity.
Hint 1
Compute step = chunk_size - overlap. Then iterate from i = 0, stepping by 'step' each time. At each position, take a slice tokens[i:i+chunk_size]. Stop once the current chunk reaches the end of the list.
Hint 2
Use a for loop: for i in range(0, len(tokens), step). Append tokens[i:i+chunk_size]. Break if i + chunk_size >= len(tokens) to avoid producing redundant trailing chunks that are entirely covered by the previous chunk.
Requirements
- Split the token list into chunks of the specified size with the given overlap between consecutive chunks
- The step between chunk start positions is chunk_size - overlap
- The last chunk may be shorter than chunk_size if there are remaining tokens
- Return a list of lists of tokens
Constraints
- tokens has at least 1 element
- chunk_size >= 1
- 0 <= overlap < chunk_size
- Return a list of lists
- Time limit: 300 ms
Try Similar Problems
Log in to take notes on this problem
Accepts: array
Accepts: number
Accepts: number