我已经阅读了很多博客,但对答案并不满意,假设我在几个文档示例上训练 tf-idf 模型:
" John like horror movie."
" Ryan watches dramatic movies"
------------so on ----------
我使用这个功能:
from sklearn.feature_extraction.text import TfidfTransformer
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(twenty_train.data)
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)
print((X_train_counts.todense()))
# Gives count of words in each document
But it doesn't tell which word? How to get words as headers in X_train_counts
outputs. Similarly in X_train_tfidf ?
所以 X_train_tfidf 输出将是具有 tf-idf 分数的矩阵:
Horror watch movie drama
doc1 score1 -- -----------
doc2 ------------------------
这样对吗?
什么
fit
做什么和做什么 transformation
做?在 sklearn 中提到:
fit(..) 方法使我们的估计器适合数据,其次是 transform(..) 方法将我们的计数矩阵转换为 tf-idf 表示。
什么
estimator to the data
方法?现在假设新的测试文件来了:
" Ron likes thriller movies"
如何将此文档转换为 tf-idf?我们不能将其转换为 tf-idf 对吗?
如何处理word
thriller
火车文件中没有。
最佳答案
以两个文本作为输入
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
text = ["John like horror movie","Ryan watches dramatic movies"]
count_vect = CountVectorizer()
tfidf_transformer = TfidfTransformer()
X_train_counts = count_vect.fit_transform(text)
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)
pd.DataFrame(X_train_tfidf.todense(), columns = count_vect.get_feature_names())
o/p
dramatic horror john like movie movies ryan watches 0 0.000000 0.471078 0.471078 0.471078 0.471078 0.335176 0.000000 0.000000 1 0.363788 0.000000 0.000000 0.000000 0.000000 0.776515 0.363788 0.363788
现在测试它的新评论,我们需要使用转换函数,在向量化时,词汇表外的单词将被忽略。
new_comment = ["ron don't like dramatic movie"]
pd.DataFrame(tfidf_transformer.transform(count_vect.transform(new_comment)).todense(), columns = count_vect.get_feature_names())
dramatic horror john like movie movies ryan watches
0 0.57735 0.0 0.0 0.57735 0.57735 0.0 0.0 0.0
如果你想使用某个单词的词汇,那么准备你想要使用的单词列表,并不断地将新单词添加到这个列表中并将列表传递给 CountVectorizer
vocabulary = ['dramatic', 'movie','horror']
vocabulary.append('Thriller')
count_vect = CountVectorizer(vocabulary = vocabulary)
cont_vect.fit_transform(text)
关于python-3.x - tf-idf 模型如何处理测试数据期间看不见的单词?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/58371573/