Implement TF-IDF Vectorizer
Implement TF-IDF Vectorizer
Implement a TF-IDF vectorizer that converts text documents into numerical feature vectors. Return both the TF-IDF matrix and the vocabulary.
Term Frequency (TF):
tf(t,d)=total terms in dcount(t,d)Inverse Document Frequency (IDF):
idf(t)=log(df(t)N)TF-IDF Score:
tf-idf(t,d)=tf(t,d)×idf(t)Where: N = total documents, df(t) = documents containing term t, count(t,d) = occurrences of t in d
Function Arguments
documents: list[str]- List of text documents to vectorize
Examples
Input: documents=["the cat sat", "the dog ran"]
Output: matrix shape (2, 5), vocab=["cat", "dog", "ran", "sat", "the"]
Input: documents=["apple banana", "banana cherry"]
Output: matrix shape (2, 3), vocab=["apple", "banana", "cherry"]
Note: "banana" appears in both docs → IDF=0, so it gets zero weight.
Hint 1
Use str.split() and str.lower() for tokenization. Use set() to build vocabulary and sorted() for consistent ordering.
Hint 2
Use Counter for term frequencies and document frequency counting. Use math.log() for IDF calculation.
Hint 3
Use np.zeros() to initialize the matrix. Create a word-to-index mapping with enumerate() for efficient matrix filling.
Requirements
- Return tuple:
(tfidf_matrix, vocabulary) tfidf_matrix:np.ndarrayof shape(n_docs, n_vocab)vocabulary:list[str]of unique terms, sorted alphabetically- Tokenize by splitting on whitespace and converting to lowercase
- Handle empty documents and empty corpus gracefully
Constraints
- Document count ≤ 10,000; vocabulary size ≤ 50,000
- Vectorized where reasonable; avoid heavy nested loops
- Libraries: NumPy + standard library only
Try Similar Problems
Log in to take notes on this problem
Accepts: array
Implement TF-IDF Vectorizer
Implement TF-IDF Vectorizer
Implement a TF-IDF vectorizer that converts text documents into numerical feature vectors. Return both the TF-IDF matrix and the vocabulary.
Term Frequency (TF):
tf(t,d)=total terms in dcount(t,d)Inverse Document Frequency (IDF):
idf(t)=log(df(t)N)TF-IDF Score:
tf-idf(t,d)=tf(t,d)×idf(t)Where: N = total documents, df(t) = documents containing term t, count(t,d) = occurrences of t in d
Function Arguments
documents: list[str]- List of text documents to vectorize
Examples
Input: documents=["the cat sat", "the dog ran"]
Output: matrix shape (2, 5), vocab=["cat", "dog", "ran", "sat", "the"]
Input: documents=["apple banana", "banana cherry"]
Output: matrix shape (2, 3), vocab=["apple", "banana", "cherry"]
Note: "banana" appears in both docs → IDF=0, so it gets zero weight.
Hint 1
Use str.split() and str.lower() for tokenization. Use set() to build vocabulary and sorted() for consistent ordering.
Hint 2
Use Counter for term frequencies and document frequency counting. Use math.log() for IDF calculation.
Hint 3
Use np.zeros() to initialize the matrix. Create a word-to-index mapping with enumerate() for efficient matrix filling.
Requirements
- Return tuple:
(tfidf_matrix, vocabulary) tfidf_matrix:np.ndarrayof shape(n_docs, n_vocab)vocabulary:list[str]of unique terms, sorted alphabetically- Tokenize by splitting on whitespace and converting to lowercase
- Handle empty documents and empty corpus gracefully
Constraints
- Document count ≤ 10,000; vocabulary size ≤ 50,000
- Vectorized where reasonable; avoid heavy nested loops
- Libraries: NumPy + standard library only
Try Similar Problems
Log in to take notes on this problem
Accepts: array