python - 使用 gensim 将 LDA 应用于语料库进行训练

Question

我的语料库包含大约 20,000 个文档，我必须使用 LDA 训练该数据集以进行主题建模。

import logging, gensim

logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
id2word = gensim.corpora.Dictionary('questions.dict')
mm = gensim.corpora.MmCorpus('questions.mm')
lda = gensim.models.ldamodel.LdaModel(corpus=mm, id2word=id2word, num_topics=100, update_every=0, chunksize=3000, passes=20)
lda.print_topics(20)

每当我运行这个程序时，我都会遇到这个错误:

2013-04-28 09:57:09,750 : INFO : adding document #0 to Dictionary(0 unique tokens)
2013-04-28 09:57:09,759 : INFO : built Dictionary(11 unique tokens) from 14 documents (total 14 corpus positions)
2013-04-28 09:57:09,785 : INFO : loaded corpus index from questions.mm.index
2013-04-28 09:57:09,790 : INFO : initializing corpus reader from questions.mm
2013-04-28 09:57:09,796 : INFO : accepted corpus with 19188 documents, 15791 features, 106222 non-zero entries
2013-04-28 09:57:09,802 : INFO : using serial LDA version on this node
2013-04-28 09:57:09,808 : INFO : running batch LDA training, 100 topics, 20 passes over the supplied corpus of 19188 documents, updating model once every 19188 documents
2013-04-28 09:57:10,267 : INFO : PROGRESS: iteration 0, at document #3000/19188

Traceback (most recent call last):
File "C:/Users/Animesh/Desktop/NLP/topicmodel/lda.py", line 10, in <module>
lda = gensim.models.ldamodel.LdaModel(corpus=mm, id2word=id2word, num_topics=100, update_every=0, chunksize=3000, passes=20)
File "C:\Python27\lib\site-packages\gensim-0.8.6-py2.7.egg\gensim\models\ldamodel.py", line 265, in __init__
self.update(corpus)
File "C:\Python27\lib\site-packages\gensim-0.8.6-py2.7.egg\gensim\models\ldamodel.py", line 445, in update
self.do_estep(chunk, other)
File "C:\Python27\lib\site-packages\gensim-0.8.6-py2.7.egg\gensim\models\ldamodel.py", line 365, in do_estep
gamma, sstats = self.inference(chunk, collect_sstats=True)
File "C:\Python27\lib\site-packages\gensim-0.8.6-py2.7.egg\gensim\models\ldamodel.py", line 318, in inference
expElogbetad = self.expElogbeta[:, ids]
IndexError: index (11) out of range (0<=index<10) in dimension 1

我什至尝试更改 LdaModel 函数中的值，但总是遇到相同的错误!

应该做什么？

Answer 1

您的词典(id2word)似乎与您的语料库对象(mm)没有正确匹配。

无论出于何种原因，id2word(单词标记到 wordid 的映射)仅包含 11 个标记 2013-04-28 09:57:09,759 : INFO : 从 14 个文档(总共 14 个语料库位置)构建字典(11 个唯一标记)

您的语料库包含 15791 个特征，因此当它查找 id > 10 的特征时，它会失败。中的 ID expElogbetad = self.expElogbeta[:, ids] 是特定文档中所有单词 ID 的列表。

我会重新运行语料库和字典的创建:

$ python -m gensim.scripts.make_wiki (来自 gensim LDA 教程)。

我认为，创建的字典的日志记录数据应该表明远远超过 11 个标记。我自己也遇到过类似的问题。