我正在使用下面的乱码评论数据在 gensim 中训练 doc2vec 模型。我遇到了 2 个错误。
1st:TaggedDocument 需要 2 个参数,我无法将 Sr
字段作为第二个参数传递,因此我采用简单的字符 ('tag')
来继续下一步。
第二:当我接近 for 循环代码末尾时,出现以下错误。
ValueError:您必须指定total_examples或total_words,才能正确更新作业参数和进度计算。通常的值为total_examples=model.corpus_count。
| Sr | review |
|------|------------------------------------------------------------|
| 123 | This is frustrating |
| 456 | I am eating in a bowl and this is frustrating |
| 678 | Summer has come and the weather is hot and I feel very hot |
| 1234 | When will winter come back I love the cool weather |
import pandas as pd
import numpy as np
import gensim
file = pd.read_csv('/Users/test_text.csv')
file1 = [line.split() for line in file.review]
sent = [gensim.models.doc2vec.TaggedDocument(lines,'tag') for lines in file1]
model = gensim.models.Doc2Vec(alpha=0.025, min_alpha=0.025,min_count=1)
model.build_vocab(sent)
for epoch in range(10):
model.train(sent)
model.alpha -= 0.002
model.min_alpha = model.alpha
最佳答案
我不知道如何用 Pandas 做到这一点。也就是说,使用 csv
模块,您可以执行以下操作:
import csv
from gensim.models.doc2vec import TaggedDocument, Doc2Vec
texts = csv.DictReader(open('test_text.csv'))
documents = [TaggedDocument(text['review'].split(), [text['Sr']]) for text in texts]
model = Doc2Vec(documents, vector_size=100, window=8, min_count=2, workers=7)
# Then you can infer new vector and compute most similar documents:
vector = model.infer_vector(['frustrating', 'bowl', 'nooddle'])
print(model.docvecs.most_similar([vector]))
它会输出如下内容:
[('123', 0.07377214729785919),
('1234', 0.019198982045054436),
('456', 0.011939050629734993),
('678', -0.14281529188156128)]
在您的情况下,数据集适合内存,因此您无需使用开始时使用的 API。
关于python - gensim 中使用 csv 的 Doc2vec,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/49526981/